GuusAssume a scenario in which a user is online with a single resource, stream management enabled, then gets disconnected abruptly, leaving the SM session in a resumable state, and a stanza arrives. What now is the expected behavior if the same user establishes a new stream, but does not resume the session but performs resource binding instead?
DanielThe sm session 'times out' all stanzas get bounced (or what ever your server is doing)
DanielWhat would the alternative be?
ZashLike, they closed their laptop, woke up their phone and left the office?
GuusWell, for stanzas that were addressed to a bare JID, it might be desirable to re-evaluate the routing logic, delivering it to the newly established session.
ZashI suspect this is a full week topic for a summit
Guusand in case the new session rebinds using the same resource... maybe deliver all stanzas to that session?
Daniel> and in case the new session rebinds using the same resource... maybe deliver all stanzas to that session?
That would maybe work for some messages. But nothing else
ZashAnd messages usually go to the bare jid now, and you can get it from MAM
DanielYes. So anything that is delivered to mam you might not want to actually bounce. I was just saying do what ever you do when you time out
Guusejabberd has an option that can be used to have all queued messages to be resend after SM times out (resend_on_timeout)
Ge0rGGuus: that's bad if you have multiple clients.
Ge0rGGuus: the existing implementations all suck
flowand potentially breaks the stable order guarantee of XMPP (depending on your PoV)
Ge0rGthe right thing to do would be to do server-side tracking of individual clients, based on their hopefully constant-over-time resource string
GuusThat's basically what I suggested, right? When resource binding on the same resource... do magic.
Ge0rGGuus: indeed, yes. But you can't know how much of the old session actually already arrived at that device, unless it attempts to resume
Ge0rGmy client won't honor the resume timeout value and just attempt to resume always, if it has a previous session
GuusGe0rG That's very true.
Guusproblem here is that this client doesn't keep track of the old session metadata if it crashes unexpectedly.
Guus(unsure if that's fixable on their end - that seems to be the preferred approach to me)
Ge0rGGuus: that's exactly what remains unsolved. Or you use MAM
jonas’better than guessing
DanielI use MAM
jonas’or churning the disk by persisting the entire stream state to disk on every. single. stanza.
Ge0rGGuus: with MAM, as Daniel said, the server should drop all message stanzas from the old session when destroying it, and not bounce them
Daniel> or churning the disk by persisting the entire stream state to disk on every. single. stanza.
I always wanted to write an iOS client that does that
Ge0rG(unfortunately, not all servers do that yet)
DanielI don't think that's too unreasonable if you plan it from the beginning and design the app around it
Ge0rGDaniel: all we need is a full redesign of the complete xmpp client stack
Ge0rGit needs to be serializable in an efficient way, ideally with some kind of journal
Ge0rGAndroid could also profit from that.
Guusflow I'm getting this from the client developers. Do you know if that's true?
> Another possible solution to the problem is to persist the stream id, session id and the number of acknowledged messages from the server and the client. This is not supported by SMACK by default,
Guusah, wait, I read that as "smack can't resume" but maybe they're saying "smack doesn't persist to disk"
DanielWell it's not as easy as storing those three values
Ge0rGGuus: yes, smack can resume, but not from disk
jonas’you also need to store all presence state
Ge0rGGuus: you'd need to serialize the XMPPTCPConnection and everything related to it to disk as well
flowbut yes, you usually want to resurrect the XMPPTCPConnection with the same listeners setup as before it was suspended to persistent storage
flowbut that does not necessarily require to serialize XMPPTCPConnection
Ge0rGno... just most of its members.
DanielWell all pending stanza, all pending callbacks to iqs
DanielLike presence, mucs etc
Ge0rGGuus: so you can't easily hibernate smack to disk and reload afterwards.
Ge0rGGuus: but you can hope that your app won't be killed
Guusand if you could, you'd have to do that preemptively on ever.single.stanza (or how did Jonas' put it?)
flowGuus, what's the problem you are trying to solve (couldn't find it in the backlog)
Ge0rGGuus: yes, after each change of the internal state (e.g. receiving or sending a stanza) you'd have to persist everything
Guusflow just a question I received over email: client gets killed, the restart it, but won't get messages that have been delivered while offline (unless they do MAM).
DanielGe0rG: well you have to persist the delta
Ge0rGDaniel: but with smack, you need to manually extract the delta from the state of dozens of weakly linked objects
DanielYeah don't use smack for that
flowGuus, use MAM ;)
GuusYeah, that's been my conclusion too. Still, I'm wondering if it makes sense to have some kind of option like ejabberd's 'resend' - but instead, I think I'd prefer to send certain messages to a new session with the same resource - something like that.
Guusprobably bad for MUC though
jonas’bad for a lot of things assuming state
flowI also think it would violate the stable order requirement of the RFC (but that may not be an issue for your use-case)
GuusI'd at the very least not enable this by default.
Guusjust trying to think of a solution that'd be helpful for certain use-cases.
Ge0rGGuus: you can't do that generally because you would end up re-sending all the stanzas that already were delivered but not acked
Ge0rGGuus: use MAM.
Ge0rGGuus: or, use eternal 0198 sessions ;)
Ge0rG(no, that doesn't solve the client crash problem)
GuusYou're right - but not every user is as obsessed with being 100.0% right all of the time as we are. 🙂
Guussome duplication might be acceptable.
GuusAnd yes, MAM is the proper way to go - but that would probably make them spend more development resources than what they have available.
Guus(afk - feeding the offspring)
jonas’Guus, some SM implementations will also feed the unacked messages to offline store when the SM session gets terminated.
jonas’that may be a viable alternative?
Ge0rGyes, if you are guaranteed to only have one client per user, you can use something like prosody's mod_smacks_offline
jonas’it’s obvious from the UI when you know what to look out for
jonas’(also, I don’t know of any end-to-end encrypted video conferencing software)
edhelaswhen you're calling yourself locally it's end to end encryted 🤔 ?
Ge0rGjonas’: that would be... interesting.
pep.I'd be interested to know how expensive that is resource-wise
pep.jitsi-meet is already getting complaints that they're not doing e2ee fwiw..
jonas’essentially you’d degrade the videobridge to a stupid TURN server
jonas’then you’re back to E2EE again
jonas’if your webrtc session does it
Ge0rGyou could do homomorphic encryption!
Ge0rGbut I suppose the more practical approach would be to encrypt the stream payloads and to let videobridge route at the stream level.
Ge0rGand to have multi-recipient keys for each stream
Ge0rGyou are typically using symmetric encryption for the data stream anyway, so just send the decryption key to all parties
Ge0rGit's not complex, just requires effort.
jonas’right, recipient keys
Ge0rGalso probably won't work with WebRTC.
Ge0rGjonas’: it depends on how you derive the session key
flowalso downscaling become
flowalso downscaling by the bridge becomes impossible with e2ee
Ge0rGwhich jitsi meet doesn't do AFAIK
Ge0rGthe right thing would be to create multiple progressive streams, where you get the lowest res with stream 0, the medium res with stream 0+1 and the highest res with 0+1+2
Ge0rGbut that would require a progressive video codec of sorts.
flowdo those exists in practice?
jonas’jitsi-videobridge (the software) does not do recoding, but there’s allegedly a hardware thing which does
NeustradamusWhat is the solution about XMPP TLS S2S problems for XMPP clients?
Example: When an firstname.lastname@example.org tries to join a email@example.com, it is not possible and the XMPP client has not an error.
MattJNeustradamus, the solution is to switch to Matrix
lovetoxdoes it make sense to flip the rsm page, and not the whole mam filter result?
lovetoxnotice this is not the same outcome
lovetoxi fail to see the use case for flipping the page
lovetoxi does not get you the filter result in the reverse order
lovetoxit does only get you the first page of the filter result, which you dont actually want, we want the last page, in reverse order
jonas’MattJ, the "Client uses two discovered query fields in a query" example differs from the previous example in the value of the @var attributes, is that intentional?
pep.MattJ, does that mean pubsub mam is going to be defined somewhere else?
ZashMaybe randomize the subdomain in `<iq to='pubsub.shakespeare.lit' type='set' id='juliet1'>` a bit more?
flowpep does it matter where it will potentially be defined?
pep.flow, it matters that it's defined, as with this PR it doesn't seem to be defined anywhere anymore
Zash> I have a few things to do before I PR (such as finishing off the other documents I split stuff out to)
pep.waiting for updates then
flowpep., was it defined before in xep313? IIRC it was just something like "ya'll can do MAM on PubSub, but I don't tell you exactly how<period>"
ZashThere's some precedent in splitting a XEP since at least MIX
Zashflow, didn't say a lot about it, but there was a few lines
pep.flow, not saying it was perfect but at least it was a thing :P
pep.I'm sure it could be defined a lot better
flowI think I would prefer if xep313 simply stated that MAM on PubSub is a possiblity but the exact specification is outside the scope if this XEP
flowhmm, not sure about that <flip-page/> thing…
flowhope it's optional
lovetoxi think i understand now the flip page thing, you are supposed to request the last page with rsm via </before> then flip that page
lovetoxand why do you hope flow its optional? are you a server developer that needs to implement that ?
lovetoxMattJ, what is the reason instead of namespace bumping this to mam:3, making it a disco feature?
flowlovetox, I consider a minimal mandatory-to-implement feature set to be benefical, as it allows to ramp up a new implementation fast
flowsure there are counterarguments, and one has to carefully select which feature to make optional
flowbut <flip-page/> appears to be a good candidate
ZashSo that's mam:2, the new stuff is mam:2#extended
pep.Zash, but it's essentially the same
pep.mam:3 or mam:2 + mam:2#extended
flow+A client that wishes to use flipped pages MUST ensure that the server
+advertises the \'urn:xmpp:mam:2\#extended\' feature.
pep.mam:2#extended is just the new mam:3
Zashpep., adding support doesn't break all existing clients
flowpep I don't think it is
Zashto the server, that is
flow#extended sounds like an optional extension to mam:2
lovetoxZash adding the extended things does not brak clients
lovetoxas it says it EXTENDS the spec
pep.It wouldn't break all clients indeed, just continue supporting mam:2 just like you're going to do anyway
lovetoxthere is no change to existing functionality
Zashlovetox, that's what I said. bumping the namespace to :3 breaks everything, unless you do both
ZashIf you have a mam:2 implementation, adding mam:2#extended doesn't affect anyone
pep.Zash, because you're still advertizing for mam:2. Which you can continue doing
lovetoxit also affects no one if you just announce mam:3
flowwell, it hopefully affects the counterparty that could talk #extended with you
MattJDoing both has turned out to be a pain i n pretty much every implementation I've seen (when we bumped other namespaces)
flowlovetox, as long as you still support mam:2
pep.MattJ, I genuinely don't get the difference
lovetoxflow its just a namespace there is nothing the server needs to do different
pep.Just that it's not explicit it's a new thing, you're just adding a new feature and now I've got to check both
lovetoxits just, if i get a IQ with mam:3 just treat it like mam:2
MattJpep.: you don't have to check both
pep.Yeah so it's just mam:3
flowhonestly, and I haven't read that diff completely so I may not seeing the whole picture, but I am surprised that the #extended thing causes any discussion
pep.Or undercovered mam:3 :P
pep.flow, to me it's not about mam, it's about how things are done in general
lovetoxlet me guess flow you never implemented mam?
flowIf the #extended thing is that I think it is, an optional extension to the mam we have, than that appears very sensible to me
flowinstead of doing mam:3
flowlovetox, I did the MAM implementation for Smack
pep.Feels like people are just afraid to increase a number
flowand wrote the unit and integration tests
flowpep., of course we are afraid
lovetoxthen i really dont get how you dont understand how this is bad
pep.Why? It's the exact same thing
lovetoxbut let me tell you why
flownamespace bumps are not for free
flowsmack is still on mam:1 fwiw
pep.flow, breaking changes are not free
lovetoxthis is not a simple syntax change in the xep, this extended functionality allows clients to do mam in a totally different way
flowpep., right that is why they have to be avoided when sensible
lovetoxits a change how archiving is done on client side in fundamental way
pep.namespace bump doesn't equal breaking changes
lovetoxto say its optional on server side
lovetoxmeans that server dont have to adopt it
lovetoxso that means i have to support mam:2 without extended
pep.Right that also..
lovetoxwhich is now not just a simple syntax change
lovetoxno i have to maintain a totally different way of doing mam
ZashHas everyone forgotten the pain of mam:0 vs mam:1 vs mam:2 so soon?
lovetoxwhat for?, these changes are trivial for servers
lovetoxthis is just some Mysql querys added differently
Zashlovetox: haha no
flowpep, I am sorry I fail to see how a namespace bump von mam:2 to mam:3 is the exact same thing as an optional extension to mam:2
pep.one could say mam:3 is an optional extension over mam:2 :)
flowmam:3 would be by definition a breaking change, as it requires a namespace bump
lovetoxflow the changes here proposed are long wished things of client developers
lovetoxthis is not some optional thing that is nice
lovetoxthese are changes the community deems necessary for an archiving xep
lovetoxthere is really no reason to make this optional
lovetoxespecially for a experimental xep
flowthere is also no reason to make it not optional?
lovetoxthe only reason something like this would be optional, if there are reasons like we believe there are implementations who simply cant implement the changes because of tecnical problems
MattJOnly that it is good features that people want to use
flowlovetox, I mean if what you and pep want is that servers are going to announce mam:2 *and* mam:3 simultanously, then I still think that this is far more effort to support than annoucning mam:2 and mam:2#extensions
flowbecause some protocol operations now need to be available under them mam:2 and mam:3 namespace
lovetoxno flow same operations can be available under both
lovetoxits no difference
lovetoxits a subset please
flowthat's what I said, some operations need to be avaialble under both namespace
lovetoxits the same operation
lovetoxdo you really think they duplicate the code because of a mam:3 string in an iq
flowno, but that does mean that it will become less ugly
flowno, but that doesn't mean that it will become less ugly
Zashlovetox, yes, bunch of duplication and more complex code
lovetoxok but thats of no concern, this is the XMPP Archving XEP this is not some sideshow where we make concessions, these are essential changes
lovetoxevery client needs that
flowthen every client will implement that
lovetoxits nothing that some weird use cases need and so we make it optional
Zashseems to me you're making very broad assumptions
lovetoxif you tell someone XMPP has a archiving xep where you cant even download a single specific message
lovetoxand thats an OPTIONAL feature, some servers implement
MattJWe're still in a world where some servers don't support it at all :)
lovetoxwe dont need to get more fragmented, we need to move faster and decisive
MattJAs far as I'm concerned, servers would be stupid not to implement the #extended stuff, but we can't force them to any more than we can force them to implement any XEP
flowlovetox, specifying something as mandatory does not mean that implementations will suddenly implement it. In fact, it will cause implementations that only implement a subset of the mandatory to implement features, which causes much more main. Features get implemented because someone decides that it is worth the effort.
flowlovetox, specifying something as mandatory does not mean that implementations will suddenly implement it. In fact, it will cause implementations that only implement a subset of the mandatory to implement features, which causes much more pain. Features get implemented because someone decides that it is worth the effort.
MattJlovetox: btw, I added page flipping for you... if you don't want it I can take it back out ;)
MattJYou and a couple of people who complained anyway
lovetoxno its good, i just wondered if its not to complicated to do the <before> and <flip-page> together
flowI'd always assume that simply requesting a small page size would be sufficient
lovetoxinstead of just reverse the filter query
flowbut I am also always in favor of experimenting with new features
MattJThere was a reason I did it on the page rather the results, I can't remember why... might be in my notes from last year somewhere
MattJThere was a reason I did it on the page rather than the results, I can't remember why... might be in my notes from last year somewhere
lovetoxprobably because of </before> in the rsm spec
lovetoxyou can already request the last page of a query
lovetoxso you just have to flip that page to finish it
MattJI think that is logically simpler anyway
lovetoxbut i guess we should mention that in the XEP
lovetoxbecause nobody will get this on their own
lovetoxit would be simpler if people would now the </before> thing in RSM
MattJThere is an example for that
lovetoxin your mam change?
lovetoxok thats nice then
MattJYes, unless it got lost
lovetoxtoo much changes, i probably missed it
MattJI'm planning a separate "How to MAM" document anyway
lovetoxwhat i want to tell is, that this is a "additional" thing on the server, you just add some different sql querys if you encounter different filters
lovetoxbut for clients its not just additional
lovetoxits either you write your impl with these futures, or without it
MattJI agree, the sensible thing is for clients to depend on #extended
lovetoxat least i believe that, have to do it :)
lovetoxok correct MattJ so from a client perspective this is the same as depending on mam:3
lovetoxso from a client perspective this is the same, and optional not useful here
lovetoxso the question remains, is it useful for the server?
MattJNot really, no
MattJI don't see any reason for people to only implement 80% of the XEP
ZashWe already implement 80% of the XEP, because those 80% was 100% of the previous version of the XEP.
MattJBut whether it is called #extended or :3 it makes no difference - people need to upgrade
jonas’the difference is that you can re-use your code from :2
MattJI mean it makes no difference to whether people implement it or not
lovetoxi dont know the process but is that code discussion really something the XSF should care about?
GuusSkipping over a lot of the conversation here: but is the flip-page directive something that adds real value? To me, it seems like a convenience for clients to not have to revert element orders if they want to. I wonder if that warrants added complexity to the protocol.
lovetoxGuus you cant invert the order
MattJGuus: that's a debate to have with lovetox and some others... you can bring it up on the list when I post it I guess
jonas’you can, you just need to buffer the (MAM) stanzas first.
jonas’but on-list is a better place for this indeed
lovetoxthats a fundamental change to how clients work jonas’
lovetoxwe dont do that for literally anything
ZashDo you know how <before/> is implemented in Prosody? ... by buffering the results and then flipping them.
MattJTo how your client works :)
lovetoxstanzas are processed when they arrive
Guusflip-page is only about order of elements in one RSM result, right?
lovetoxGuus a RSM page has usually 100 stanzas
flowHmm, then shouldn't something like flip-page be part of RSM instead?
Guus... not seeing the problem - unless the RSM pages are ungodly huge.
Guusso - reduce the RSM size?
lovetoxthen mam catchup is much slower ?
flowGuus, I think something like flip-page really adds value over low latency connections
MattJflow: I considered that, but I don't want to bump RSM ;)
flowMattJ, optional extension?
ZashIf you're doing infinite scroll and scroll up?
GuusOkay, if it has real world benefits I'm happy to include it. Seems far-fetched to me, but then again, I'm no client builder 🙂
lovetoxalso this means i have to download a 100 stanzas, before i can show the user the first
lovetoxseems like just not a good protocol
jonas’slow connections is a valid point
GuusNah, just limit the max result set page size to 20 or somesuch?
lovetoxthen you wait for 20 ..
jonas’in other RSM cases it’s not as relevant (cc @ MattJ) because normally you get an RSM result page in a single stanza
flowerr, I meant high latency connections obviously
GuusThe entire point of having a page is that it offers a manageable bite size of data.
jonas’Guus, then you have expensive round-trips. Flipping the result makes a lot of sense here.
jonas’MAM is a splendid bulk operation from the client side
flowbut thinking about it a little more, I am actually not that sure if it adds value
flowthose 20 stanzas are all bundled
jonas’Guus, especially on long-fat-links (high latency, lots of throughput) you want few roundtrips and lots of data. quite a bit of mobile links these days are like that, especially when you’re on the road: your packets are buffered somewhere and then you get a bunch of them back. roundtrips are expensive.
flowand what's typical stanzas size? how long does it really take to receive 20 stanzas?
jonas’flow, they don’t fit in a single TCP segment, so it can take arbitrarily long for the last (first) stanza to be delayed while you already have 19 others
flowtrue, if you not only have high latency but also packet loss
jonas’or irregular high latency
jonas’as you have on mobile links
GuusI'm not dead-set against this - but how is this now a problem, and not in the past 10 years? Connectivity has improved, generally...
jonas’you sometimes see sudden RTTs of 60s!
jonas’Guus, it’s always been a problem
flowas I said, I think <flip-page/> is a prime candidate of an optional feature. Not everyone may want or needs to implement it
jonas’clients are just "crap" and sync from oldest messages first and not from newest first
jonas’probably also because of stuff like this
flowGuus, I'd argue bad connectiviy is probably more widespread
jonas’when you sync from oldest, it doesn’t matter, but it’s also terrible UX
flowif you think about all those smartphones in africe
flowif you think about all those smartphones in africa
Guusoh well. I'll take your word for it 🙂
jonas’once you’re in an intercity train
Guusno wait, my parents' wifi.
MattJAs far as I'm concerned it adds minimal value, but it makes lives for some people easier, so I'm fine with including it
lovetoxSorry this is a messaging protocol and we discuss if its too much to ask from a archive to get the messages in the order you want them
lovetoxdownload the archive and reverse them yourself
MattJlovetox: you just want SQL replication over XMPP ;)
Guusno, we're discussing if adding a feature that I thought was trivial to implement in code warrants added complexity in the protocol
jonas’MattJ, don’t confuse lovetox with zinid
Guusthat is a relevant discussion, especially in a standards body.
ZashMattJ: You mean XEP-0136
lovetoxGuus, its the simple use case, of loading messages while you scroll upwards in your messenger
lovetoxjust think from a client point of view how you want to implement that
jonas’MattJ, is the auto-pull of xeps working again?
lovetoxthis is not someting exotic, ANY user expect the client to do this
lovetoxin a efficient fashion
Guuslovetox as I already said: I'm happy to accept that this is less trivial as what I thought it was, going into this discussion.
MattJMost XMPP alternatives will be using HTTP and JSON. Trust me, they don't start displaying results until they receive the full result set from the server
jonas’MattJ, but that also means that they can handle latency spikes much better, since they have regular TCP restarts instead of having to deal with a TCP recovering from a minute of packet loss
ZashDid someone say BOSH?
MattJjonas’: I'd be surprised to see anyone using a TCP connection per request these days
jonas’MattJ, I wouldn’t be surprised for them restarting a fresh TCP connection when the other one runs in a timeout
jonas’when you do that with XMPP, you’re in a world of massive pain
jonas’unless you’re very clever, but nobody wants to be that these das
jonas’unless you’re very clever, but nobody wants to be that these days
jonas’(start a new session, SM-resume your other, suppesedly timed-out stream with your current local counters, discard the other stream once you got the resumption ACK)
flowahh muc-presence-versioning now in html, nice
flowhmm, or not?
jonas’I pushed the docker image, no idea if the pull is working
jonas’hence 18:36:31 jonas’> MattJ, is the auto-pull of xeps working again?
pep.flow, "specifying something as mandatory does not mean that implementations will suddenly implement it" I totally agree. And I don't think this applies "we shall then make features optional". At least making something mandatory would tell implementors that these features are important
jonas’MattJ, can you issue a manual pull then please?
jonas’or, has board already decided about my iteam membership? ;)
MattJWhen I get back to my laptop, in an hour or however long it takes children to go to bed
jonas’pep., I’m more in favour of having documents which give a rationale for the importance of a feature, instead of just relying on the MUST hammer
jonas’say hello from the strange internet parrot people
flowpep., why not simply tell the implementors that these features are important instead?
pep.Well that's how you tell them it's important
pep."Ah btw that's what 90% of clients are going to use nowadays, but if you don't implement it it's fine don't worry"
pep.Is how I see the optional thing
flowpep., again, then why not simply tell the implementors that these feature are important to preven this?
flowpep., again, then why not simply tell the implementors that these feature are important to prevent them from getting that impression?
pep.Suggestion for this year's CS, add namespaces to be supported. So that people don't claim they support mam:0 (or whatever NS that nobody uses anymore) and be done with it
lovetoxand then add it to the compliance suite :D
lovetoxif i think about thats actually what i care for
lovetoxi want the server to know what they should implement
lovetoxand thats my fear that optional things get only implemented if i personally bug every server dev
lovetoxso yes if we have a document that tells the server dev what he should actually implement to have a relevant impl, then i guess im fine with the XEP not mandating it
jonas’in contrast to that, I bet you’ll find every mandatory thing of '45 or '60 implemented in all servers
flowotherwhise your fear would be that that big monohilitc everything-is-mandatory spec does not get implemented at all, or only partly (which would cause a lot of "fun")
pep.Well quite a lot of MUSTs or SHOULDs are behind optional features
jonas’discoverable partial support is better than only finding out about lack of support the hard way
pep.flow, it's fine if the thing is still usable
pep.But it's useless if it isn't
jonas’which can either be a <service-unavailable/> / <feature-not-implemented/> (if you’re lucky), or different behaviour because an element was ignored.
flowpep., I am sorry pep, but I can't follow
pep."I implemented all the mandatory bits! No clients can use my implementation!"
pep.(all and only*)
Douglas Terabytehas left
Douglas Terabytehas joined
NeustradamusTwo "Version 1.0.0 of XEP-0363" in next Newsletter