XSF Discussion - 2018-08-20

Does anyone remeber what the point of https://xmpp.org/extensions/xep-0292.html#self-iq was?

What's wrong with that? MUCs are made to scale!

Ge0rG: but my eyes don't scale 😞

Maranda: are you reading the XML log?

...and only see the blondes, the brunettes, ...

Ge0rG: trying to catch something from the log tail actually 🙄☺️

depending on the CSI implementation and violation of RFCs.

Although staying in CSI theme...

"For example: A client sends a CSI active nonza, followed by an XMPP Ping request to the server. The server first changes the CSI state to active and flushes all eventually queued *stanzsa*."

Maranda: but you are dropping stanzas

Ge0rG, so whenever I filter a stanza silently I'm breaking in order processing...?

Perhaps everyone violates rfcs

Ge0rG, Does the RFC forbid that?

The RFC states that stanza order MUST be preserved.

Yep, but I meant "dropping stanzas"

Ge0rG, so since *everyone* does that I guess there isn't a real problem isn't it? :P

dropping != reordering, I believe.

or you can't filter and drop stanzas anymore

Guus, I'm not reordering

I'm not saying you are. 🙂

:)

I do wonder how effective CSI can be though, given that there's bound to be a stanza that you don't want to queue, and you need to maintain stanza order.

doesn't give much room to play, I think

at least not for "drop all but the last" type of filtering.

Guus, I'm just queuing presences so it's... rather easy to do

Maranda, sure, but as soon as you get an IQ ping, you'd have to flush all that, right? Before responding to the IQ ping, I mean.

(or any other non-queued stanza)

Guus depends on who send the ping. but yes, usually you would flush if you have to send something anyway

Guus, hmm that I didn't think about.

flow, an inbound ping to the client, I mean. I don't think it matters much who sent the ping? The server needs to preserve the order of all stanzas, not depending on who is the originator?

So you want to tell me it is okay with the RFC to drop stanzas, but not ok to reorder them?

Guus, I believe that XMPP could fall into the category where the order only matters with regard to two endpoints, not necesarily globally

flow: I'm not sure if that works. How would that work with private messages in MUCs, for example

So if the ping originated from an entity which has no queued stanzas, you could only deliver the ping without flushing (but there is no reason you shouldn't flush)

Ge0rG: I'm not sure, to be honest. My point is even if it were allowed, I wonder if there's much practical benefit to CSI.

got to pick up kid from school. bbl

Guus: flow: in practical terms the order between two bare JIDs must be preserved

Ge0rG: but isn't that how privacy lists work too, btw?

MattJ, why bare JID and not just "JID"?

flow: imagine reordering presence from a MUC join, which specifies you always receive your own presence last

In my opinion servers need to be very careful if they start arbitrarily dropping or reordering stanzas

What MattJ said.

well if it's bare jids IQs are excluded from the equation I suppose.

clients won't have to answer to those.

Maranda: I didn't mean it like that

Reordering is worse than not delivering ✏

But I'm on my phone so I don't think I can attempt a better explain

(Note that 'not delivering' isn't the same as 'dropping' - if you don't deliver, you still have to follow the rules for when to silently drop and when to error)

My point is: if the RFC mandates in-order delivery, it is kind of implicit that it mandates delivery.

So just dropping stanzas will violate the RFC

... and so everyone in a way or another violates it.

Ge0rG: Delivery or error, yes.

It's my understanding that it mandates delivery

Or error

Some cases allow silent dropping.

But XEPs can override core rules?

Zash: yes

Depending on stanza type, as you well know

Zash: Yes, by negotiation.

MattJ, yep, the xep45 self-presence order is a good point

Perhaps you're suggesting to do some magic amendment to the CSI spec?

:P

Maranda: the CSI spec does not tell you to reorder or drop

Maranda, some ppl will tell you that you violate the RFC, some will tell you possible the opposite. What matters is: Is it beneficial? Does it break things?

Unfortunately it has some "examples" which I am not happy about, but consensus was to add them

They are not part of CSI

And they are not specified in any detail, which may be worse

Then get rid of them from the XEP. But we should consider adding a wiki page for CSI where strategies are collected

Working and non-working strategies that is

Pros and cons

flow: iirc you were one of the main voices requesting their addition :)

And whys

flow, it's very beneficial and doesn't break anything tbh.

Maranda: how do you know it is beneficial?

MattJ, deduplicating presences?

This is another problem I have with current CSI usage

MattJ, well it reduces queues *a lot* expecially if you join a lot of big mucs.

Maranda: OK, spammy broken clients

But the MUC could filter that

More simply

MattJ, irregardless of broken clients

bbl, train arriving at station

MattJ, wait, we are talking about the in order requirement § 5.1? I thought we talk about CSI strategies that are applied

flow, I guess that if you queued all stanzas there wouldn't be problem with in order processing, the issues come out only if you don't filter all routable payloads but only one or two of 'em. ✏

(like I currently do)

Maranda, figures would be interesting. like the bytes that are not send by your implementation

Guus: > I do wonder how effective CSI can be though, given that there's bound to be a stanza that you don't want to queue, and you need to maintain stanza order. For me in practice there's often long periods where many presence stanzas are received, and nothing else.

flow, well for my mobile account the average queue for staying inactive few hours was like 600 presences without dedup, it went down to as little as 50-60. ✏

Maranda, so that is CSI with and without queue (presence) dedup?

So I guess that matters somehow, about the actual bw you save I'm not sure but you can make an esteem.

yes

k, I was more wondering about CSI vs non-CSI

so there's for sure a benefit expecially on MUC usage.

and especially the longest time period the connection was idle

time for mod_csi_stats

flow: Dedup can of course make a great difference if you limit the queue size and flush when it's reached.

well I suppose you can extract those numbers from my figure if the server queued around 600 presences mostly for mucs during inactive hours ✏

for sure the reduction is reasonably around x10 magnitude

sending only the last useful presence state

Holger, good point

heck, it doesn't need MUCs - I have an account with 700 transport contacts from a network with lousily defined presence semantics and CSI helps there as well in taming the unnecessary traffic

dos, which network is this?

facebook :P

even grouping the presence XML stanzas over time and sending them in one big flush is good for the battery.

Ge0rG, yes which is where deduplication helps...

> <MattJ> flow: imagine reordering presence from a MUC join, which specifies you always receive your own presence last I guess the problem with breaking this is that the client might mis-interpret presence received after the final self-presence as *newly* joined members?

Yes

dos: any actual stats?

Join a few noicy IRC channels

Holger: imagine how that interacts with GC1.0 joins and intermittent s2s outages.

There's no GC1.0 la-la-la can't hear you

I've done a CSI benchmark on a freenode transport some two years ago.

I prepared a motion to Council to get rid of GC1.0, but it was not approved.

And when I started making a PR to 0045, I realized that significant deletion is required

Next Prosody won't allow GC1.0 joins, have we seen any ill effects of it yet?

So you basically cannot de-dup MUC presence without breaking that 0045 guarantee unless you add special code for handling MUC joins.

Holger: Mmmmm, special cases

They're so elegant!

MattJ: I haven't made any stats so far, but when I hacked up my Nokia N900 to actually use CSI, it was the first time I could actually observe prolonged silence on the stream

Specal casing on top of hacks on top of more hacks until your brain explodes

I've got around 60 presences in the last few minutes

fwiw I didn't yet see a single muc break.

Maranda: Do you have a CSI client that shows MUC joins/leaves?

Without one I'm not sure how you'd "see" that breakage.

and even then it might actually be hard to notice

Holger, join a small channel compare occupant list with another client?

you're likely to join MUCs when CSI indicates "active", for instance

Maranda: The resulting occupant list will be correct in both cases.

dos: another good one is client caps caching

I'm sure that works, and I just did that because I was worried something like that happened because of deduplication.

Holger, I didn't mean breaking the spec :P

Maranda: AFAICS the only problem would be bogus "Maranda joined" messages.

I meant *it actually* broke anything.

dos: my client will join MUCs when inactive, because a MUC ping timeout happened when idle

Ge0rG: yup. thankfully, that was already done by the whichevet old Telepathy version is there :)

Maranda: Yes I think it doesn't *actually* break anything because Conversations & friends don't show "Maranda joined" messages anyway.

Holger, I'd suppose clients look more at status 110 now than the order.

for self presence

Ge0rG: and how likely are you to notice some erroreneous joins displayed by your client in such case? ;D

Maranda: That statement makes no sense to me.

dos: there are issues.

Maranda: The point is, when they receive their status-110-message they assume they have the full list of occupants. Any presence received afterwards would be interpreted as "Maranda joined", while in reality Maranda didn't join but was an occupant before I joined.

Am I being completely daft, or does https://tools.ietf.org/html/rfc6121#section-8.5 completely fail to say how to route presence type=error?

Holger, hm ok now that's clearer :P

I was more looking at actual occupant list desyncs rather than that anyways

Yes those shouldn't be broken by this kind of dedup.

only temporarily

Well yes only until the client processed the flushed stanzas.

or if your dedup has a bug

:-)

I haven't heard of bugs in XMPP code.

that’s because they eat the messages telling you about them

Holger, but question is how many mobile clients (one presumes most using CSI are those) actually show MUC joins? ✏

Maemo 5 does, and I modified it to use CSI, so I guess I could test it with ejabberd :v

dos: doo eet 😁

Maranda: Yes, as I said :-)

Maranda: The answer may well be zero.

Maranda: I'm doing that kind of dedup as well as the possible benefit seems significant to me, and I haven't seen any (reports on) breagage. But this *is* hairy of course.

"breagage", I'm drunk.

Holger, ditto I mean as far as I like strict compliance, the current CSI code works and is (surprisingly) clean and simple for once, so as long as something doesn't break (e.g. like occupant list desync) I'm not inclined to touch it 😁

MattJ, could you kick anjan temporarily? apparently, they’re flooding this room with useless <presence/> stanzas

:)

An unsurprising outcome, in hindsight

GC1.0 FTW!

Unfortunately, there is no way to burn GC1.0 without "breaking" usability.

damn I actually needed anjan to test something

Maranda: get yourself kickbanned as well, then you are together with anjan again 🤔

Ge0rG, :P

Ge0rG, no need it actually works.

And that problem wasnt gc1.0 related me thinks just an old issue in gajim that made it loop resending muc presences

Maranda: but with GC1.0, the ghost returned after getting kicked

🤔

Btw, are there any docs on GC1.0? We didn't find it anywhere, only references in XEP-0045

hmm... when client receives a message directed to other client via carbon that has a delivery receipt request, should it send one?

That can make sense

nvm, xep mentions it, so I guess it should: "Because it is possible for a given content message to be delivered to multiple XMPP resources controlled by the recipient, the sender of the content message needs to be prepared to receive multiple ack messages."

The xep is probably not talking about carbons though

But still I belive sending them - while not ideal - is the best thing we can do for now

I think this is the only mention of a situation with multiple acks, so I'd interpret it as "it is possible, for instance with bare JID messages, message carbons are enabled etc."

so nothing really mandates you to send them, but conforming receipt requestor should be prepared to handle them, so it should be fine

I'd rather send it. Even if sender didn't intend my device to receive this message, I still received it, because my carbons are configured this way. So, it's delivered to me, and I should probably send a receipt.

I also don't like when remote party dictates how my devices work. "I'll talk with you over this Android phone, not over that desktop client"

#OMEMO

you could send a carbon of a receipt 😁

Psi does it even better right now - it sends a receipt for an outgoing carbon that requests one :D

Heh.

dos: as long as it doesn't request a receipt for a receipt, everything is good.

Something something two generals problem

in my opinion of course a receipt should be sent ✏

a carbon means only the server received the message, not the client

lovetox: what?

what part of what i said is not clear?

> a carbon means only the server received the message, not the client

maybe I'm just lacking the proper context for this

you receive a "received" carbon, now you can wait 5 seconds or any amount of time you want, if there is no receipt coming in from your other client, you should assume he didnt receive the message

and ack

I'd just ack anyway

ack for carbons, ack for mam, ack all teh time

we need multi-ack!

if we only had a mechanism for acknowledging stuff

like, IQs

or send acks when the server writes into mam

Zash, I’d like that

wait have we invented email then?

i think thats exactly not what receipts are about

receipt mean the client received the message

not a server archive, which cant even determine if the message is encrypted and the client can decrypt it

really there are error messages for when s2s or c2s does not work

except when there aren’t

what if we just stop worrying about everything

messages get through or they don't

yes i say this often, Gajim has receipts disabled by default

i use xmpp now for some years, never had a problem with not received messages, except when encryption was involved

.I agree with lovetox in that.

sure, let’s not worry about reliability

lovetox, I had

lovetox: sorry, but your story is a huge anomaly. I lose messages all the time

a lot

sometimes I only lose carbons because my contact just turns off their phone's wifi

did you say something?

but many of those would probably be solved by MAM

Ge0rG: maybe you smacks implementation is broken?

Or the fact that you don't have mam?

daniel: maybe XMPP is broken?

Apparently not for me and lovetox

Who do implement both mam and 198

daniel: not sure about ejabberd, but another widely-deployed XMPP server will error-bounce all stanzas when a hibernated 0198 session gets destroyed.

besides, I probably just don't know zilch about how xmpp works :P

But a bounced message is not a lost one

Unless you don't track your errors

Schrödingers bouncy deliveries

afaik ejabberd has a config entry for that and can either put them to offline storage or bounce

Yeah so does prosody. But complaining about things is always more fun

The day humans stop complaining is the day we gone extinct.

daniel: the prosody module doesn't work if you have multiple clients

But yeah, complaining is more fun.

Also I'm not sure what tracking errors gives you, unless you do automatic retransmission, which is a huge PITA

We need some ELE first or later anyways

ELE?

jonasw: Extinction Level Event

Maranda: https://syonyk.blogspot.com/2015/08/i-am-backsideist.html

sounds like some complex read which I'm a bit too tired to start :O

> not sure about ejabberd, but another widely-deployed XMPP server will error-bounce all stanzas when a hibernated 0198 session gets destroyed. well I have the "Ge0rG's maneuver" when there's at least one session online supporting carbons :P

(aka "assume like it's delivered")

(... and don't bounce it)

Maranda: is that upstreamed?

Ge0rG, it's in Metronome from quite a bit yes.

Maranda: that's not what I mean.

Then I'm too tired to understand what you mean, elaborate plx :P

Maranda: prosody

And I don't think so, dunno.

> is quite unpleasantly high, the state of infrastructure is quite sad (when was the last bridge collapse?), 6 days ago.

Not speaking of irc bridges