XSF Discussion - 2020-03-31

Assume a scenario in which a user is online with a single resource, stream management enabled, then gets disconnected abruptly, leaving the SM session in a resumable state, and a stanza arrives. What now is the expected behavior if the same user establishes a new stream, but does not resume the session but performs resource binding instead?

The sm session 'times out' all stanzas get bounced (or what ever your server is doing)

What would the alternative be?

Like, they closed their laptop, woke up their phone and left the office?

Well, for stanzas that were addressed to a bare JID, it might be desirable to re-evaluate the routing logic, delivering it to the newly established session.

I suspect this is a full week topic for a summit

and in case the new session rebinds using the same resource... maybe deliver all stanzas to that session?

> and in case the new session rebinds using the same resource... maybe deliver all stanzas to that session? That would maybe work for some messages. But nothing else

And messages usually go to the bare jid now, and you can get it from MAM

Yes. So anything that is delivered to mam you might not want to actually bounce. I was just saying do what ever you do when you time out

ejabberd has an option that can be used to have all queued messages to be resend after SM times out (resend_on_timeout)

Guus: that's bad if you have multiple clients.

Guus: the existing implementations all suck

and potentially breaks the stable order guarantee of XMPP (depending on your PoV)

the right thing to do would be to do server-side tracking of individual clients, based on their hopefully constant-over-time resource string

That's basically what I suggested, right? When resource binding on the same resource... do magic.

Guus: indeed, yes. But you can't know how much of the old session actually already arrived at that device, unless it attempts to resume

my client won't honor the resume timeout value and just attempt to resume always, if it has a previous session

Ge0rG That's very true.

problem here is that this client doesn't keep track of the old session metadata if it crashes unexpectedly.

(unsure if that's fixable on their end - that seems to be the preferred approach to me)

Guus: that's exactly what remains unsolved. Or you use MAM

use MAM

better than guessing

I use MAM

or churning the disk by persisting the entire stream state to disk on every. single. stanza.

Guus: with MAM, as Daniel said, the server should drop all message stanzas from the old session when destroying it, and not bounce them

> or churning the disk by persisting the entire stream state to disk on every. single. stanza. I always wanted to write an iOS client that does that

(unfortunately, not all servers do that yet)

I don't think that's too unreasonable if you plan it from the beginning and design the app around it

Daniel: yes

Daniel: all we need is a full redesign of the complete xmpp client stack

it needs to be serializable in an efficient way, ideally with some kind of journal

Android could also profit from that.

flow I'm getting this from the client developers. Do you know if that's true? > Another possible solution to the problem is to persist the stream id, session id and the number of acknowledged messages from the server and the client. This is not supported by SMACK by default,

ah, wait, I read that as "smack can't resume" but maybe they're saying "smack doesn't persist to disk"

Well it's not as easy as storing those three values

Guus: yes, smack can resume, but not from disk

you also need to store all presence state

Guus: you'd need to serialize the XMPPTCPConnection and everything related to it to disk as well

depends, no?

but yes, you usually want to resurrect the XMPPTCPConnection with the same listeners setup as before it was suspended to persistent storage

but that does not necessarily require to serialize XMPPTCPConnection

no... just most of its members.

Well all pending stanza, all pending callbacks to iqs

All state

Like presence, mucs etc

Guus: so you can't easily hibernate smack to disk and reload afterwards.

Guus: but you can hope that your app won't be killed

and if you could, you'd have to do that preemptively on ever.single.stanza (or how did Jonas' put it?)

Guus, what's the problem you are trying to solve (couldn't find it in the backlog)

Guus: yes, after each change of the internal state (e.g. receiving or sending a stanza) you'd have to persist everything

flow just a question I received over email: client gets killed, the restart it, but won't get messages that have been delivered while offline (unless they do MAM).

Ge0rG: well you have to persist the delta

Daniel: but with smack, you need to manually extract the delta from the state of dozens of weakly linked objects

Yeah don't use smack for that

Guus, use MAM ;)

Yeah, that's been my conclusion too. Still, I'm wondering if it makes sense to have some kind of option like ejabberd's 'resend' - but instead, I think I'd prefer to send certain messages to a new session with the same resource - something like that.

probably bad for MUC though

bad for a lot of things assuming state

I also think it would violate the stable order requirement of the RFC (but that may not be an issue for your use-case)

I'd at the very least not enable this by default.

just trying to think of a solution that'd be helpful for certain use-cases.

Guus: you can't do that generally because you would end up re-sending all the stanzas that already were delivered but not acked

Guus: use MAM.

Guus: or, use eternal 0198 sessions ;)

(no, that doesn't solve the client crash problem)

You're right - but not every user is as obsessed with being 100.0% right all of the time as we are. 🙂

some duplication might be acceptable.

(to them)

And yes, MAM is the proper way to go - but that would probably make them spend more development resources than what they have available.

(afk - feeding the offspring)

Guus, some SM implementations will also feed the unacked messages to offline store when the SM session gets terminated.

that may be a viable alternative?

yes, if you are guaranteed to only have one client per user, you can use something like prosody's mod_smacks_offline

https://theintercept.com/2020/03/31/zoom-meeting-encryption/

ah that

it’s obvious from the UI when you know what to look out for

(also, I don’t know of any end-to-end encrypted video conferencing software)

when you're calling yourself locally it's end to end encryted 🤔 ?

jonas’: that would be... interesting.

I'd be interested to know how expensive that is resource-wise

jitsi-meet is already getting complaints that they're not doing e2ee fwiw..

essentially you’d degrade the videobridge to a stupid TURN server

then you’re back to E2EE again

if your webrtc session does it

you could do homomorphic encryption!

but I suppose the more practical approach would be to encrypt the stream payloads and to let videobridge route at the stream level.

and to have multi-recipient keys for each stream

you are typically using symmetric encryption for the data stream anyway, so just send the decryption key to all parties

it's not complex, just requires effort.

right, recipient keys

also probably won't work with WebRTC.

it should be possible on the protocol level, but I wouldn’t be surprised if javascript APIs don’t expose this

jonas’: it depends on how you derive the session key

also downscaling by the bridge becomes impossible with e2ee ✏

which jitsi meet doesn't do AFAIK

the right thing would be to create multiple progressive streams, where you get the lowest res with stream 0, the medium res with stream 0+1 and the highest res with 0+1+2

but that would require a progressive video codec of sorts.

do those exists in practice?

jitsi-videobridge (the software) does not do recoding, but there’s allegedly a hardware thing which does

What is the solution about XMPP TLS S2S problems for XMPP clients? Example: When an user@xmpp1.tld tries to join a mucroom@conference.xmpp2.tld, it is not possible and the XMPP client has not an error.

Neustradamus, the solution is to switch to Matrix

:D

Prosody will be a Matrix server?

Who knows

https://jabber.fu-berlin.de/share/holger/xczTzkOs6Bduvkve/Awe5-jk3SCOXiFasIOrOag.jpg

Conversations says "remote server not found" if I try to join mucroom@conference.xmpp2.tld.

Holger, in case you missed the previous discussion on this topic, you probably don't want to reopen this can of worms

😂

Holger: The log on muc.xmpp.org: - User has joined - User has left Client has not error and the client always tries but it has already failed.

https://i.imgflip.com/3usu50.jpg

Ge0rG: example jabber.org (not only) -> xmpp.org is broken.

New draft MAM revision currently here: https://github.com/mwild1/xeps/tree/mam-r7

I'll post to the list about it soon (probably tomorrow)

I have a few things to do before I PR (such as finishing off the other documents I split stuff out to)

https://cerdale.zash.se/upload/DCzM-vgHHb2_CbHA/diff-mam-r7.html

nice MattJ

does it make sense to flip the rsm page, and not the whole mam filter result?

notice this is not the same outcome

i fail to see the use case for flipping the page

i does not get you the filter result in the reverse order

it does only get you the first page of the filter result, which you dont actually want, we want the last page, in reverse order

MattJ, the "Client uses two discovered query fields in a query" example differs from the previous example in the value of the @var attributes, is that intentional?

MattJ, does that mean pubsub mam is going to be defined somewhere else?

Maybe randomize the subdomain in `<iq to='pubsub.shakespeare.lit' type='set' id='juliet1'>` a bit more?

pep does it matter where it will potentially be defined?

flow, it matters that it's defined, as with this PR it doesn't seem to be defined anywhere anymore

> I have a few things to do before I PR (such as finishing off the other documents I split stuff out to) pep. ↑

ok

waiting for updates then

pep., was it defined before in xep313? IIRC it was just something like "ya'll can do MAM on PubSub, but I don't tell you exactly how<period>"

There's some precedent in splitting a XEP since at least MIX

flow, didn't say a lot about it, but there was a few lines

flow, not saying it was perfect but at least it was a thing :P

I'm sure it could be defined a lot better

I think I would prefer if xep313 simply stated that MAM on PubSub is a possiblity but the exact specification is outside the scope if this XEP

hmm, not sure about that <flip-page/> thing…

hope it's optional

i think i understand now the flip page thing, you are supposed to request the last page with rsm via </before> then flip that page

and why do you hope flow its optional? are you a server developer that needs to implement that ?

MattJ, what is the reason instead of namespace bumping this to mam:3, making it a disco feature?

lovetox, I consider a minimal mandatory-to-implement feature set to be benefical, as it allows to ramp up a new implementation fast

sure there are counterarguments, and one has to carefully select which feature to make optional

but <flip-page/> appears to be a good candidate

So that's mam:2, the new stuff is mam:2#extended

Zash, but it's essentially the same

mam:3 or mam:2 + mam:2#extended

+A client that wishes to use flipped pages MUST ensure that the server +advertises the \'urn:xmpp:mam:2\#extended\' feature.

mam:2#extended is just the new mam:3

pep., adding support doesn't break all existing clients

pep I don't think it is

to the server, that is

#extended sounds like an optional extension to mam:2

Zash adding the extended things does not brak clients

as it says it EXTENDS the spec

It wouldn't break all clients indeed, just continue supporting mam:2 just like you're going to do anyway

there is no change to existing functionality

lovetox, that's what I said. bumping the namespace to :3 breaks everything, unless you do both

If you have a mam:2 implementation, adding mam:2#extended doesn't affect anyone

Zash, because you're still advertizing for mam:2. Which you can continue doing

it also affects no one if you just announce mam:3

well, it hopefully affects the counterparty that could talk #extended with you

Doing both has turned out to be a pain i n pretty much every implementation I've seen (when we bumped other namespaces)

lovetox, as long as you still support mam:2

MattJ, I genuinely don't get the difference

flow its just a namespace there is nothing the server needs to do different

Just that it's not explicit it's a new thing, you're just adding a new feature and now I've got to check both

its just, if i get a IQ with mam:3 just treat it like mam:2

finished

pep.: you don't have to check both

Just one

Yeah so it's just mam:3

No

honestly, and I haven't read that diff completely so I may not seeing the whole picture, but I am surprised that the #extended thing causes any discussion

Or undercovered mam:3 :P

flow, to me it's not about mam, it's about how things are done in general

let me guess flow you never implemented mam?

If the #extended thing is that I think it is, an optional extension to the mam we have, than that appears very sensible to me

instead of doing mam:3

lovetox, I did the MAM implementation for Smack

Feels like people are just afraid to increase a number

and wrote the unit and integration tests

pep., of course we are afraid

then i really dont get how you dont understand how this is bad

Why? It's the exact same thing

but let me tell you why

namespace bumps are not for free

smack is still on mam:1 fwiw

flow, breaking changes are not free

this is not a simple syntax change in the xep, this extended functionality allows clients to do mam in a totally different way

pep., right that is why they have to be avoided when sensible

its a change how archiving is done on client side in fundamental way

namespace bump doesn't equal breaking changes

to say its optional on server side

means that server dont have to adopt it

so that means i have to support mam:2 without extended

Right that also..

which is now not just a simple syntax change

no i have to maintain a totally different way of doing mam

Has everyone forgotten the pain of mam:0 vs mam:1 vs mam:2 so soon?

what for?, these changes are trivial for servers

this is just some Mysql querys added differently

lovetox: haha no

pep, I am sorry I fail to see how a namespace bump von mam:2 to mam:3 is the exact same thing as an optional extension to mam:2

one could say mam:3 is an optional extension over mam:2 :)

mam:3 would be by definition a breaking change, as it requires a namespace bump

flow the changes here proposed are long wished things of client developers

this is not some optional thing that is nice

these are changes the community deems necessary for an archiving xep

there is really no reason to make this optional

especially for a experimental xep

there is also no reason to make it not optional?

the only reason something like this would be optional, if there are reasons like we believe there are implementations who simply cant implement the changes because of tecnical problems

Only that it is good features that people want to use

lovetox, I mean if what you and pep want is that servers are going to announce mam:2 *and* mam:3 simultanously, then I still think that this is far more effort to support than annoucning mam:2 and mam:2#extensions

because some protocol operations now need to be available under them mam:2 and mam:3 namespace

no flow same operations can be available under both

its no difference

its a subset please

that's what I said, some operations need to be avaialble under both namespace

its the same operation

do you really think they duplicate the code because of a mam:3 string in an iq

no, but that doesn't mean that it will become less ugly ✏

lovetox, yes, bunch of duplication and more complex code

ok but thats of no concern, this is the XMPP Archving XEP this is not some sideshow where we make concessions, these are essential changes

every client needs that

then every client will implement that

its nothing that some weird use cases need and so we make it optional

seems to me you're making very broad assumptions

if you tell someone XMPP has a archiving xep where you cant even download a single specific message

and thats an OPTIONAL feature, some servers implement

thats insane

We're still in a world where some servers don't support it at all :)

we dont need to get more fragmented, we need to move faster and decisive

As far as I'm concerned, servers would be stupid not to implement the #extended stuff, but we can't force them to any more than we can force them to implement any XEP

lovetox, specifying something as mandatory does not mean that implementations will suddenly implement it. In fact, it will cause implementations that only implement a subset of the mandatory to implement features, which causes much more pain. Features get implemented because someone decides that it is worth the effort. ✏

lovetox: btw, I added page flipping for you... if you don't want it I can take it back out ;)

You and a couple of people who complained anyway

no its good, i just wondered if its not to complicated to do the <before> and <flip-page> together

I'd always assume that simply requesting a small page size would be sufficient

instead of just reverse the filter query

but I am also always in favor of experimenting with new features

There was a reason I did it on the page rather than the results, I can't remember why... might be in my notes from last year somewhere ✏

probably because of </before> in the rsm spec

you can already request the last page of a query

Yeah

so you just have to flip that page to finish it

I think that is logically simpler anyway

but i guess we should mention that in the XEP

because nobody will get this on their own

it would be simpler if people would now the </before> thing in RSM

There is an example for that

in your mam change?

ok thats nice then

Yes, unless it got lost

too much changes, i probably missed it

I'm planning a separate "How to MAM" document anyway

what i want to tell is, that this is a "additional" thing on the server, you just add some different sql querys if you encounter different filters

but for clients its not just additional

its either you write your impl with these futures, or without it

I agree, the sensible thing is for clients to depend on #extended

at least i believe that, have to do it :)

ok correct MattJ so from a client perspective this is the same as depending on mam:3

so from a client perspective this is the same, and optional not useful here

so the question remains, is it useful for the server?

Not really, no

I don't see any reason for people to only implement 80% of the XEP

We already implement 80% of the XEP, because those 80% was 100% of the previous version of the XEP.

Obviously

But whether it is called #extended or :3 it makes no difference - people need to upgrade

the difference is that you can re-use your code from :2

I mean it makes no difference to whether people implement it or not

i dont know the process but is that code discussion really something the XSF should care about?

Skipping over a lot of the conversation here: but is the flip-page directive something that adds real value? To me, it seems like a convenience for clients to not have to revert element orders if they want to. I wonder if that warrants added complexity to the protocol.

Guus you cant invert the order

Guus: that's a debate to have with lovetox and some others... you can bring it up on the list when I post it I guess

you can, you just need to buffer the (MAM) stanzas first.

but on-list is a better place for this indeed

thats a fundamental change to how clients work jonas’

we dont do that for literally anything

Do you know how <before/> is implemented in Prosody? ... by buffering the results and then flipping them.

To how your client works :)

stanzas are processed when they arrive

flip-page is only about order of elements in one RSM result, right?

Guus: yes

Guus a RSM page has usually 100 stanzas

Hmm, then shouldn't something like flip-page be part of RSM instead?

... not seeing the problem - unless the RSM pages are ungodly huge.

so - reduce the RSM size?

then mam catchup is much slower ?

Guus, I think something like flip-page really adds value over low latency connections

flow: I considered that, but I don't want to bump RSM ;)

MattJ, optional extension?

If you're doing infinite scroll and scroll up?

Okay, if it has real world benefits I'm happy to include it. Seems far-fetched to me, but then again, I'm no client builder 🙂

also this means i have to download a 100 stanzas, before i can show the user the first

seems like just not a good protocol

slow connections is a valid point

Nah, just limit the max result set page size to 20 or somesuch?

then you wait for 20 ..

in other RSM cases it’s not as relevant (cc @ MattJ) because normally you get an RSM result page in a single stanza

err, I meant high latency connections obviously

The entire point of having a page is that it offers a manageable bite size of data.

Guus, then you have expensive round-trips. Flipping the result makes a lot of sense here.

jonas’: yeah

MAM is a splendid bulk operation from the client side

but thinking about it a little more, I am actually not that sure if it adds value

those 20 stanzas are all bundled

Guus, especially on long-fat-links (high latency, lots of throughput) you want few roundtrips and lots of data. quite a bit of mobile links these days are like that, especially when you’re on the road: your packets are buffered somewhere and then you get a bunch of them back. roundtrips are expensive.

and what's typical stanzas size? how long does it really take to receive 20 stanzas?

flow, they don’t fit in a single TCP segment, so it can take arbitrarily long for the last (first) stanza to be delayed while you already have 19 others

true, if you not only have high latency but also packet loss

or irregular high latency

as you have on mobile links

I'm not dead-set against this - but how is this now a problem, and not in the past 10 years? Connectivity has improved, generally...

you sometimes see sudden RTTs of 60s!

Guus, it’s always been a problem

as I said, I think <flip-page/> is a prime candidate of an optional feature. Not everyone may want or needs to implement it

clients are just "crap" and sync from oldest messages first and not from newest first

probably also because of stuff like this

Guus, I'd argue bad connectiviy is probably more widespread

when you sync from oldest, it doesn’t matter, but it’s also terrible UX

if you think about all those smartphones in africa ✏

or germany

oh well. I'll take your word for it 🙂

once you’re in an intercity train

#stayhome

(problem solved)

no wait, my parents' wifi.

As far as I'm concerned it adds minimal value, but it makes lives for some people easier, so I'm fine with including it

Sorry this is a messaging protocol and we discuss if its too much to ask from a archive to get the messages in the order you want them

seriously lol

download the archive and reverse them yourself

ok

lovetox: you just want SQL replication over XMPP ;)

no, we're discussing if adding a feature that I thought was trivial to implement in code warrants added complexity in the protocol

MattJ, don’t confuse lovetox with zinid

that is a relevant discussion, especially in a standards body.

MattJ: You mean XEP-0136

Guus, its the simple use case, of loading messages while you scroll upwards in your messenger

just think from a client point of view how you want to implement that

MattJ, is the auto-pull of xeps working again?

this is not someting exotic, ANY user expect the client to do this

in a efficient fashion

lovetox as I already said: I'm happy to accept that this is less trivial as what I thought it was, going into this discussion.

Most XMPP alternatives will be using HTTP and JSON. Trust me, they don't start displaying results until they receive the full result set from the server

MattJ, but that also means that they can handle latency spikes much better, since they have regular TCP restarts instead of having to deal with a TCP recovering from a minute of packet loss

Did someone say BOSH?

jonas’: I'd be surprised to see anyone using a TCP connection per request these days

MattJ, I wouldn’t be surprised for them restarting a fresh TCP connection when the other one runs in a timeout

when you do that with XMPP, you’re in a world of massive pain

unless you’re very clever, but nobody wants to be that these days ✏

(start a new session, SM-resume your other, suppesedly timed-out stream with your current local counters, discard the other stream once you got the resumption ACK)

ahh muc-presence-versioning now in html, nice

hmm, or not?

I pushed the docker image, no idea if the pull is working

hence 18:36:31 jonas’> MattJ, is the auto-pull of xeps working again?

It's not

flow, "specifying something as mandatory does not mean that implementations will suddenly implement it" I totally agree. And I don't think this applies "we shall then make features optional". At least making something mandatory would tell implementors that these features are important

MattJ, can you issue a manual pull then please?

or, has board already decided about my iteam membership? ;)

When I get back to my laptop, in an hour or however long it takes children to go to bed

pep., I’m more in favour of having documents which give a rationale for the importance of a feature, instead of just relying on the MUST hammer

MattJ, kthx

say hello from the strange internet parrot people

:)

pep., why not simply tell the implementors that these features are important instead?

Well that's how you tell them it's important

"Ah btw that's what 90% of clients are going to use nowadays, but if you don't implement it it's fine don't worry"

Is how I see the optional thing

pep., again, then why not simply tell the implementors that these feature are important to prevent them from getting that impression? ✏

Suggestion for this year's CS, add namespaces to be supported. So that people don't claim they support mam:0 (or whatever NS that nobody uses anymore) and be done with it

and then add it to the compliance suite :D

if i think about thats actually what i care for

i want the server to know what they should implement

and thats my fear that optional things get only implemented if i personally bug every server dev

yep

so yes if we have a document that tells the server dev what he should actually implement to have a relevant impl, then i guess im fine with the XEP not mandating it

in contrast to that, I bet you’ll find every mandatory thing of '45 or '60 implemented in all servers

hah

otherwhise your fear would be that that big monohilitc everything-is-mandatory spec does not get implemented at all, or only partly (which would cause a lot of "fun")

Well quite a lot of MUSTs or SHOULDs are behind optional features

discoverable partial support is better than only finding out about lack of support the hard way

flow, it's fine if the thing is still usable

But it's useless if it isn't

which can either be a <service-unavailable/> / <feature-not-implemented/> (if you’re lucky), or different behaviour because an element was ignored.

pep., I am sorry pep, but I can't follow

"I implemented all the mandatory bits! No clients can use my implementation!"

(all and only*)

MattJ, poke

Two "Version 1.0.0 of XEP-0363" in next Newsletter

jonas’, done

k