-
SamWhited
Looks like this co-op magazine about the web might be taking submissions if anyone is a good writer and wants to make a few extra dollars while writing about XMPP: https://compost.digital/
-
MattJ
SamWhited: > Does it matter if you include <after> in Result Set management, but also set after-id in the MAM form? Does the latest ID win? The form is the filter query, RSM is for paging through the results of the query.
-
jonas’
form is where, RSM is limit✎ -
jonas’
form is WHERE, RSM is LIMIT ✏
-
Zash
And OFFSET
-
jonas’
yep
-
Zash
Except how do you offset on an id?
-
jonas’
yes, it’s better than SQL in that regard
-
jonas’
but semantically it’s that
-
SamWhited
RSM has an 'after' concept too though.
-
Zash
SamWhited: I think you misunderstood what was said. The RSM 'after' corresponds to OFFSET in SQL
-
SamWhited
I don't see how that's different than. after-id still
-
Zash
after-id is WHERE RSM after is OFFSET
-
jonas’
SamWhited, that `after` in MAM-RSM corresponds to the stanza ID is an implementation detail you must not rely on
-
jonas’
on the surface it doesn’t matter, but there are ugly edge cases and long mailing list discussions which came to be because of the lack of a WHERE which is separate from the RSM-<after/>/<before/>-LIMIT mechanism
-
Zash
The MAM stuff selects a subset of messages. RSM shifts a view over that subset.
-
Zash
FWIW while implementing extended MAM in Prosody I cheated and just put the after-id in the same slot in the internal API as the RSM after
-
Zash
But, implementation detail..
-
jonas’
shush!
-
Zash
Only matters for <count> I think, which should not be affected by RSM parameters, only MAM ones
-
SamWhited
ooh, I need to read RSM again I think then. Are you saying that 'after' is an int, not an ID?
-
jonas’
SamWhited, no, after is an opaque string
-
jonas’
you mustn’t interpret it (on the client side anyway)
-
SamWhited
I still have zero concept of what you mean by 'offset' and 'where' then
-
Zash
SamWhited: I think we were talking abstract concepts, not exact SQL mapping
-
jonas’
I was for sure
-
Zash
OFFSET in SQL is an int, yes. But conceptually you SELECT a range, and then shift the starting point with OFFSET
-
jonas’
no, you select a *subset of records*
-
jonas’
not necessarily a range (which is thought of to be regular or consecutive even)
-
Zash
/correct s/range/subset/
-
Zash
I must have thought of a SELECT with only after-id
-
MattJ
Step 1 is to run the query and get the results, step 2 is to return the correct page. If you use <after> in RSM with an id that isn't in the result set you'll get an item-not-found: https://xmpp.org/extensions/xep-0059.html#notfound
-
MattJ
(previous versions of 313 indicated otherwise, but this was fixed in one of the later revisions)
-
SamWhited
Okay, no longer on a phone and coffee has been aquired, I was hoping that would help, but I still don't see how that's different from after-id. I mean, they both contain "after" in the name and take an ID, they both return an error if the ID doesn't exist.
-
SamWhited
Going back to read the RSM description of "after" now.
-
Kev
It’s not a mechanical objection, AFAICT Sam, but a case of dogma not wanting to use RSM for this. Obviously it worked mechanically using RSM, because that was deployed and working.
-
SamWhited
Kev: I guess that's what I don't get then, it says "use RSM" but then also says "Also here's more form fields that appear the same"
-
SamWhited
Is it just that RSM requires max (I think?) and this is just "after but without requiring a limit"?
-
SamWhited
Or is it that RSM doesn't specify what the ID is and it might be an ID local to the result set and not the actual stanza ID like after-id takes? Although in that case I'd question why it didn't just say "use RSM, and use the stanza ID"
-
Zash
Isn't that exactly what MAM says?
-
Zash
You use RSM, IDs are stanza-ids.
-
SamWhited
Okay, then it definitely seems like exactly the same thing in two places in the sstanza to me
-
SamWhited
Maybe someone can provide a SQL query that includes <after> and after-id how they would translate?
-
Zash
Again, conceptually, after-id would map to `WHERE item_id > after-id` and RSM after would map to `OFFSET after-first.id` or something (excuse the math)
-
SamWhited
So after-id is some sort of string comparison? I dont' get that from reading the XEP
-
SamWhited
Nothing says that IDs have to be ordered as far as I can see
-
Zash
You're still taking this too literally
-
SamWhited
Okay, let's step back then, I don't understand what the difference between WHERE and OFFSET is in that query if it's not "one is doing a string comparison, one is saying 'give me rows that are after this row'"
-
Zash
So yeah in actual SQL it'd be more like `WHERE _internal_id > (SELECT _internal_id WHERE id=$after_id}`
-
Zash
Unless you actually do have ordered IDs
-
SamWhited
That seems *exactly* the same as offset then
-
Zash
But we can't count on that, it's an implementation detail
-
Zash
It's not
-
Zash
You could implement it either way, but it's not the same, conceptually.
-
SamWhited
I guess I don't see how. Or if they are the same but just "different conceptually" why bother having both?
-
SamWhited
Let me try this: would I ever actually *use* both in a MAM query?
-
Zash
You could also implement it by not having any limit, collect *all* the results and then throw away items until you go past the RSM after id
-
Zash
Then the MAM fields go in the SQL, RSM go into cropping the results afterwards
-
SamWhited
Where does it say that?
-
Zash
Using both after-id and rsm after would be ponitless yes.
-
SamWhited
So what is the point of having both if using them both would be pointless?
-
SamWhited
Or does it actually specify how I should implement it somewhere?
-
Zash
It's written a bit up where MattJ said so.
-
SamWhited
That's just an error that gets returned?
-
Zash
> Step 1 is to run the query and get the results, step 2 is to return the correct page.
-
Zash
That part
-
SamWhited
Where does the XEP say that though?
-
SamWhited
And why would it force an implementation detail like that on me?
-
SamWhited
Eg. wouldn't I just have a single concept of "give me stuff after this ID" and then if I wanted to just directly query for that vs. query for everything, cache those results, and return subsets of them as more queries come in entirely my decision and I wouldn't need two afters to indicate either thing?
-
MattJ
Step 1: you want to query the archive between two ids Step 2: You formulate the query using after-id and before-id Step 3: You send the query to the server Step 4: The server returns the results, but there are many, so it returns the first page of 20 stanzas Step 5: You make a second request with the same query parameters, but specify <after>last-id-in-the-page</after> in your RSM payload
-
MattJ
You *could* just update after-id in the query parameters, you'll get the same messages
-
SamWhited
Why not just make the query specifying <after> to start with, is what I'm asking?
-
edhelas
"how to mimic SQL requests on top of a protocol where the server is actually doing SQL requests behind"
-
MattJ
after-id didn't always exist, and it has different semantics (e.g. you can't specify before+after in RSM)
-
MattJ
But yes, they do overlap, but they are conceptually different
-
Zash
It matters more for before, since that implicitly makes it so you start at the end of the results and page backwards
-
SamWhited
I don't see why MAM couldn't just say "if before and after are in RSM it's a range in this spec", but okay, that's the first thing that's vaguely sounded like a real difference to me, thanks
-
MattJ
So I recommend using each one for its intended purpose
-
SamWhited
ahhh, okay, before meaning you go backwards makes sense, so it would need a different concept I guess. That's absolutely infuriating, but makes sense.
-
SamWhited
Okay, I dunno, I guess I still don't see why MAM couldn't just specify that, but I guess it's a tiny bit weird modifying RSM behavior just for one spec. This seems like a major failure of RSM though that we had to include another concept of the exact same thing, but also sort of use RSM but only sometimes, etc.
-
MattJ
I think RSM is pretty good at what it's designed for
-
MattJ
It's not designed for querying
-
SamWhited
Fair enough; seems like it just shouldn't be used in MAM at all then.
-
MattJ
and the alternative would be duplicating all the paging syntax and semantics into MAM
-
SamWhited
Maybe I'm really the only one who's confused by this, but that would seem to be better. If RSM just doesn't work for what MAM's trying to do, shoehorning it in and duplicating some of its concepts just seems confusing to me.
-
MattJ
It does work
-
MattJ
RSM is not for querying, MAM is not for paging
-
MattJ
I really don't see the problem :)
-
MattJ
Unless you choose to confuse the two
-
SamWhited
That seems like a distinction without a difference to me in this particular case.
-
MattJ
If you choose to treat them as the same thing, then you're inviting confusion in edge cases
-
SamWhited
I dunno, maybe I'm just being obtuse since no one else seems to have a problem with this, but saying "there's after and there's after, they do the same thing but one of them is for querying initially and one is for subsequent queries to get pages later" just seems insane
-
MattJ
I don't see a problem with "here is a query", and "here is how you access further pages of the same query"
-
SamWhited
But again, I seem to be the only one confused by this so I guess I'm wrong 🤷\
-
MattJ
You decide what results you are interested in, and then you page through until you're done
-
SamWhited
And the spec says *none* of this as far as I can tell. There was one throw away comment about how after/before behavior wasn't defined in RSM, that's it I think
-
SamWhited
The spec just leaves you wondering why after-id and after aren't the same thing.
-
MattJ
Says none of what exactly?
-
MattJ
"A client or server will typically want to limit the number of results transmitted at a time, thereby breaking the result stream into smaller 'pages'. For this purpose a server MUST support Result Set Management (XEP-0059) [4] and MUST support the paging mechanism defined therein."
-
MattJ
That's the section that introduces RSM
-
SamWhited
Doesn't say that RSM is for querying after the first query and after-id is for the first query or whwhatever
-
MattJ
Because that's not the case
-
SamWhited
It just introduces two concepts, after and after-id that at first glance appear identical.
-
SamWhited
I swear that's literally what you just said, no?
-
MattJ
RSM is not designed for querying
-
MattJ
Yes, you can get the same result set two different ways (after-id or using RSM) in specific query types
-
MattJ
But you're designing an API here
-
MattJ
So this should be of benefit to you, you need to get it right
-
Zash
grep vs more ?
-
SamWhited
"filtering vs. querying" then. The point is that the spec just says "you can use RSM, it has after, there is also after-id in the form" (not an actual quote, obviously).
-
MattJ
1. choose the results you are interested in 2. select the page
-
MattJ
In an API these things should be 100% different
-
MattJ
For some queries (where you only care about ids), sure, you could do everything with RSM
-
MattJ
For anything more complex, they are not the same thing
-
SamWhited
I'm not sure if this is offtopic or not, because I honestly can't tell how it applies to after-id vs. after. But when zash says "grep vs more" or MattJ says "1. choose the results you are interested in 2. select the page", does it matter that I'm doing that behind thes scenes? I'm not sure how to phrase this, but that just sounds like an implementation detail that doesn't matter.
-
SamWhited
I would probably *never* do that in an implementation. I would always just make small queries, return results. Those results include info for making the next query. I would never actually pull out the entire archive within a range and then allow smaller queries to be filtered on that.
-
SamWhited
I can't tell if it actually matters for this discussion though.
-
MattJ
Just make your API nice, I don't care what mechanics you use if it works and is a nice API
-
MattJ
If you do it wrong I just reserve the right to never look at the code :)
-
SamWhited
I *think* my API is nice, but as far as I can tell no matter how complex the query you could set the ID in after or after-id *unless* you need a window and want to set after/before. So the XMPP API still seems really weird to me.
-
SamWhited
(assuming you're using an ID and want stuff after it at all, I mean, obviously you could also set neither for the initial query)
-
SamWhited
So I can do what you say and use after-id the first time and <after> afterwards, it just seems worth clarifying in the XEP that you should always do that.
-
MattJ
You don't just use after-id the first time, you can include it every time
-
SamWhited
Oh crap, okay, now I'm back to square one and have no idea what you're talking about or why I'd do that.
-
MattJ
Ok, if we're back to square one then I have better things to do :)
-
MattJ
Just implement it, I'm sure you'll manage
-
SamWhited
Maybe it really is just me in which case fair enough, but I really don't see how this distinction isn't confusing and doesn't need to be clarified in the XEP.
-
MattJ
Also consider whether you want to make RSM reusable in your codebase
-
MattJ
Because it sounds like you're not even considering that
-
SamWhited
RSM is already resuable in the codebase.
-
MattJ
But you don't want to reuse it for some reason, even though MAM explicitly allows it
-
MattJ
You would rather recalculate MAM query parameters instead of just tacking on the RSM page request
-
SamWhited
I mean, I want to reuse it if it fits, but it sounds like MAM is having to partially re-invent it which is just confusing.
-
MattJ
I don't care what you do, but anything else seems more work
-
MattJ
MAM is not partially reinventing it because RSM does *not* do querying
-
MattJ
If you read the XEP that's pretty clear
-
SamWhited
I don't have to recalculate anything, it sounds like *exactly* the same thing to me which is what I still don't get. You make a query, get results back. You make another query and change <after> (or after-id) to the last one you got from the previous query.
-
MattJ
The RSM XEP, that is
-
SamWhited
MAM you have to make a query. RSM picks the page you get. I get that, I just don't see how it changes that after and after-id are the same thing.
-
MattJ
If you start with a time range query you're not just changing after-id, you're adding it. And if you want the last page first, what are you going to do then?
-
MattJ
Yes, I 100% agree that in some cases you can achieve the same query in two different ways
-
MattJ
I 100% don't think that this is something that needs to take up an afternoon of discussion
-
MattJ
If your API has the concept of a query, and the concept of a page, they map perfectly to MAM and to RSM
-
SamWhited
> If you start with a time range query you're not just changing after-id, you're adding it. Sure, but that would be the exact same thing still if you use after-id or <after>, right? Yes, you can also make other queries, I don't see how that changes anything. > > And if you want the last page first, what are you going to do then? You make the query using <before/> just like both MAM and RSM say to do, it has nothing to do with after/after-id being duplicated?
-
MattJ
if you want to conflate the two, feel free to do that
-
SamWhited
Okay, I dunno, I don't think I'm conflating anything and I don't see why that distinction should even exist, which is what's concerning me, but I'll go re-read the relavant sections of MAM again and see if I can find anthing that clarifies this that I might have forgotten. Thanks for your help.
-
SamWhited
This is just driving me *nuts* because I don't understand the miscommunication. I'm not sure if I'm asking something wrong or there's some assumption others have that I don't, or what. It literally seems like we just have two identical things in the XEP and the only reason is that *technically* there is some slightly semantic difference in the words we use to describe them but no practical difference.
-
Zash
I couldn't consume coffee fast enough to keep up.
-
SamWhited
Okay, sorry, last thing just to make sure I'm being absolutely clear about what's confusing me with a practical example, then I'll just give up and go re-read MAM again:
-
SamWhited
Is there any difference in results that should be returned for the following two queries: https://gist.github.com/SamWhited/bdc7b2465a5f91d92bc881a3367d97d3
-
Zash
SamWhited: Same messages, but they might differ on <rsm:count>
-
SamWhited
ahhhh, okay, that's the difference, I understand now I think.
-
SamWhited
There is actually a thing that would change, not just some vague semantic difference in wording but in practice they're identical.
-
SamWhited
I am not sure what I should have asked to get at that sooner, but maybe the lesson is "always include examples".
-
Zash
Since that count is the count of the whole set, if no RSM or limits were involved. But it's also not guaranteed to be exact.
-
MattJ
or present
-
Zash
Examples good.
-
SamWhited
Sure, I definitely wouldn't include that (it's very expensive in postgres without lots of hackery), but at least there is a real practical difference between the two concepts now.
-
MattJ
There already was (besides the few offered earlier)
-
Zash
I think in Prosody we only include it if you also ask for rsm:max=0
-
MattJ
Anyway, back to work
-
SamWhited
Going back I don't see how I would have gotten there from any of the previous replies, they all seem to sound like "sure, they're identical, but we can use different words to describe them"
-
SamWhited
I'm sure I was jus tasking something weird though and not making the confusion clear.
-
SamWhited
(ignoring the before/after thing, that one I agree is vaguely a reason, it just also seems like it would have been easy enough to do in MAM)
-
Kev
It is possible that not using RSM at all would be sensible. It would certainly be possible.
-
Kev
It seemed like a good idea at the time, though.
-
SamWhited
To be clear, I am not advocating for a specific solution. I wasn't even sure there was a problem. I was just trying to understand why both after/after-id existed (or, with the reason I was given, why it actually matters). I think "count would be different" is the reason.
-
SamWhited
MattJ: are there any pending TODOs on MAM? Would you mind if the editor did another LC?
-
MattJ
Nothing pending, no
-
SamWhited
Would you mind if a LC happened then? There is one "FIXME" in the text that I can propose an editorial change for after work, but otherwise I'd like to see it advance if possible.
-
SamWhited
(unless of course you'd prefer to do it as the author, I just thought I'd offer)
-
MattJ
Nope, go for it
-
SamWhited
Thanks
-
SamWhited
/cc whomever the editor is these days. jonas’ maybe?
-
MattJ
The last revision was intended to be the last revision, I just wanted to wait a while for feedback
-
MattJ
It's been a while, so I'm happy with moving it on now
-
Kev
Maybe I should actually read the latest version :D
-
jonas’
SamWhited, ACK, maybe ask for LC together with your PR is easiest✎ -
SamWhited
MattJ: that's sort of what I was thinking. Given how small and mostly-inactive this community is, I figure just doing a LC is probably the best way to actually get people to look at it :)
-
jonas’
SamWhited, ACK, maybe ask for LC together with your PR that should be easiest ✏
-
jonas’
Kev, well, that’s what the LC is for :)
-
MattJ
I also gave plenty of opportunity for feedback before submitting the revision too, so another argument for moving ahead
-
SamWhited
Will do.
-
jonas’
SamWhited, thanks!
-
Kev
jonas’: Well, kinda. I’m Author too ;)
-
Zash
Haha
-
jonas’
Kev, are you sure? ;)
-
Kev
:(
-
arc
Meeting time?
-
jonas’
arc, wrong day of week?
-
jonas’
or did board reschedule?
-
arc
We rescheduled
-
MattJ
Wait what
-
jonas’
oh, was that not announced on board@?
-
jonas’
oh, in the minutes
-
MattJ
I guess I should have read those more thorougly
-
MattJ
It's very unlikely I'll be able to attend
-
arc
I would happily be wrong on this
-
ralphm
We could skip though, since we're only half a week on
-
arc
I'm cool with that
-
ralphm
MattJ, does the new slot work in general?
-
arc
I just got out of the ER with extremely high blood pressure
-
ralphm
Dude. Take care of yourself first.
-
ralphm
I make it motion if necessary✎ -
ralphm
I'll make it a motion if necessary ✏
-
jonas’
floor comment: I support that motion
-
ralphm
Motion carries :D
-
jonas’
ralphm, MR 20201126T15:04:59Z 000 <MattJ> I generally have a hard cut-off at 1700Z in winter, 1600Z in summer
-
jonas’
followed-up by: MR 20201126T15:07:53Z 000 <MattJ> If we must, I can probably do that on a Thursday or Friday, at least during the winter
-
ralphm
Sorry my exocortex didn't raise that
- ralphm not banging the gavel for the Board Meeting => +1W
-
MattJ
I can't do this time on any Wednesday and sensibly participate. My wife is working and I look after the children 😕
-
ralphm
Ok. Back to the drawing board then
-
emus
Have you tried using those appointment finding polls?
-
DebXWoody
OpenPGP for XMPP - Context IM Is there a valid use case where a user has two OpenPGP keys which both are associated with one XMPP account?
-
vanitasvitae
I mean, the spec does not forbid it
-
vanitasvitae
DebXWoody ^
-
vanitasvitae
So why noz
-
vanitasvitae
I think thats partly why there is the metadata node with public key list: https://xmpp.org/extensions/xep-0373.html#announcing-pubkey-list
-
DebXWoody
(on one device ). To have more keys in general (multi device) yes. But about more than one key in one local keyring.
-
vanitasvitae
I think there is no point in that. Then again I see no interop-concerns if a device has two secret keys. There might be a use-case for that as well (different keys presented to different contacts, like a work-key and a free-time key?)
-
vanitasvitae
But other devices would simply see two keys as if they would belong to different devices as there is no binding between devices and keys
-
vanitasvitae
So if your client wants to allow two secret keys on a device, I see no issues with that 😀
-
DebXWoody
I won't 😆 The user has to choose which key he will use to sign a message. I was thinking to show an error, but I don't know of there may some use cases.
-
vanitasvitae
if I'd implement OX, I'd either generate a key for the user, or if the user can provide their own keys, I'd ask them for the key-ID they want to use 🙂
-
vanitasvitae
btw. DebXWoody are you subscribed to standards@?
-
vanitasvitae
I just dropped a mail asking for possible dates for a regular OX meeting 😉