jdev - 2020-08-11

mod_smacks did have such limits for a short time, but they caused exactly this problem and were then removed until someone can come up with a better way to deal with it

was that the thing that made my server explode due to an unbound smacks queue? ;)

I got the impression that yesterdays discussion was about the opposite problem, killing the session once the queue is too large

So, no, not that issue.

it turned out the user which reported that problem had a queue size of 1000

what lovetox describes sounds like a case of too much burst traffic, not of socket synchronization issues

the current ejabberd default is much higher

but a few versions ago it was 1000

if you join a dozen MUCs at the same time, you might well run into a 1000 stanza limit

1000 is like nothing

you cant even join one irc room like #ubuntu or #python

lovetox: only join MUCs once at a time ;)

you get instantly disconnected

Matrix HQ

unless the bridge is down ;)

the current ejabberd default is 5000

which until now works ok

I'm sure the Matrix HQ has more users. But maybe it's slow enough in pushing them over the bridge that you can fetch them from your server before you are killed

Ge0rG: Oh, you're suggesting that it's a kill based on timing out an ack because the server is ignoring that it's reading stuff from the socket that's acking old stanzas, rather than timing out on data not being received?

That seems plausible.

Kev: I'm not sure it has to do with the server's reading side of the c2s socket at all

Dunno how ejabberd works but that old mod_smacks version just killed the session once it hit n queued stanzas.

Holger: I'm not sure I underestood it either

Kev: I think it's about N stanzas suddenly arriving for a client, with N being larger than the maximum queue size

Zash: That's how ejabberd works. Yes that's problematic, but doing nothing is even more so, and so far I see no better solution.

Ah. I understand now, yes.

Holger: you could add a time-based component to that, i.e. allow short bursts to exceed the limit

give the client a chance to consume the burst and to ack it

I mean if you do nothing it's absolutely trivial to OOM-kill the entire server.

Holger: BTDT

Ge0rG: Wasn't that a feedback loop tho?

Zash: yes, but a queue limit would have prevented it

It's not clear to me how you solve that problem reasonably.

Keep an eye on the largest known MUCs, and make the limit slightly larger than the sum of the top 5 room occupants

And by MUCs, I also mean bridged rooms of any sort

Get rid of stream management :-)

Get rid of the queue

Get rid of clients

I think this is only related to stream management, no? You end up with a queue somewhere?

Yes! Only servers!

Zash, peer to peer?

NOOOOOOOO

Holger, Zash: we could implement per-JID s2s backpressure

Well no, bu tyes

Kev: You end up with stored MAM messages.

s2s 0198, but scoped to individual JIDs

also that old revision that allowed to request throttling from the remote end

You could make it so that resumption is not possible if there's more unacked stanzas than a (smaller) queue size

At some point it's going to be just as expensive to start over with a fresh session

Zash: a client that auto-joins a big MUC on connect will surely cope with such invisible limits

Where you obviously might want to implement some sort of disk storage quota, but that's less likely to be too small for clients to cope. Also the burst is often just presence stanzas, which we might be able to reduce/avoid some way.

Soooo, presence based MUC is the problem yet again

Anyway, until you guys fixed all these things for me, I'll want to have a queue size limit :-)

I remember discussing MUC optimizations, like skipping most initial presence for large channels

we need incremental presence updates.

ejabberd's room config has an "omit that presence crap altogether" knob. I think p1 customers usually press that and then things suddenly work.

isn't there a XEP for room presence list deltas

I also don't enjoy getting megabytes of presence upon joining all the MUCs

eta: Yeah, XEP-0436 MUC presence versioning

does anyone plan on implementing it?

I suspect someone is. Not me tho, not right now.

Having experimented with presence deduplication, I got the feeling that every single presence stanza is unique, making deltas* pretty large ✏

oh gods

And given the rate of presence updates in the kind of MUC where you'd want optimizations... not sure how much deltas will help.\

Yeah I was wondering about the effectiveness for large rooms as well.

Just recording every presence update and replaying it like MAM sure won't do. Actual diff will be better, but will it be enough?

Would be nice to have some kind of numbers

So we need to split presence into "room membership updates" and "live user status updates"?

MIX?

Affiliation updates and quitjoins is easy enough to separate

and then we end up with matrix-style rooms, and some clients joining and leaving the membership all the time

So we have affiliations, currently present nicknames (ie roles) and presence updates

I've been thinking along the lines of that early CSI presence optimizer, where you'd only send presence for "active users" (spoke recently or somesuch). Would be neat to have a summary-ish stanza saying "I just sent you n out of m presences"

You could also ignore pure presence updates from unaffiliated users and that kind of thing

also you only want to know the total number of users and the first page full of them, the other ones aren't displayed anyway ;)

Yeah

Zash> Soooo, presence based MUC is the problem yet again I think the fundamental design problem is pushing stanzas instead of recipients requesting them. Think for example a participant of a high traffic MUC using a low throughput connection (e.g. GSM). That MUC could easily kill the participants connection

You do request them by joining.

Zash, sure, let me clarify: requesting them in smaller batches (e.g. MAM pagination style) ✏

You just described how Matrix works btw

I did not know that, but it appears like one (probably sensible) solution to the flow control / traffic management problem we have

or like MIX ;D

let's just do everything in small batches.

correct me if I am wrong, but MIX's default modus operandi is still to fan-out all messages

I think only if you subscribe to messages

also, I thought we were talking about *presence*, not messages.

I think the stanza kind does not matter

if someone sends you stanzas with a higher rate than you can consume some intermedidate queue will fill

yeah, well, that’s true for everything

hence I wrote "fundamental design problem"

I can see the case for MUC/MIX presence because that’s a massive amplification (you send single presence, you get a gazillion and a continuous stream back)

yeah, no, I don’t believe in polling for messages

The main issue is catchup.

if you’re into that kind of stuff, use BOSH

I did not say anything about polling

Whether when you join you receive a flood of everything, or whether you request stuff when you're ready for it, in batches.

Using MAM on MIX is meant to give you the latter.

and yes, the problem is more likely caused by presence stanzas, but could be caused by IQs or messages as well

If you have a room that is itself generating 'live' stanzas at such a rate that it fills queues, that is also a problem, but is distinct from the 'joining lots of MUCs disconnects me' problem.

Kev, using the user's MAM service or the MIX channel's MAM service?

Both use the same paging mechanic.

12:41:06 flow1> Zash, sure, let me clarify: requesting them in smaller batches (e.g. MAM pagination style) how is that not polling then?

though I sense that this is a discussion about semantics I don’t want to get into right now.

right, I wanted to head towards the question on how to be notified that there are new messages that you may want to request

by receiving a <message/> with the message.

that does not appear to be a solution, as you easily run into the same problem

[citation needed]

MAM/Sub!

I was thinking more along the lines of infrequent/slightly delayed notifications with the current stanza/message head ID(s) ✏

but then again, it does not appear to be a elegant solution (or potentially is no solution at all)

Oh, this is basically the same problem as IP congestion, is it not?

And the way to solve that is to throw data away. Enjoy telling your users that.

> The main issue is catchup. This. So now you'll have to figure out what data got thrown away and fetch it.

(Also how Matrix works.)

the one thing that may be good to steal from matrix is push rules

i.e. some server side filtering you can do to figure out what should generate a push notification

Can you rephrase that in a way that doesn't make me want to say "but they stole this from us"

well so CSI filtering is an XMPP technology, right

but there's no API to extend it

like you can't say "please send me everything matching the regex /e+ta/"

"push rules" meaning what, exactly?

Zash: it's just reusing good ideas :p

You said "push notifications", so I assumed "*mobile* push notifications"

Zash: a filter that the client can define to tell the server what's "important"

AMP?

Zash, so yeah, push rules are used for mobile push notifications in Matrix

Push a mod_firewall script? 🙂

for push notifications, the logic is in the push server, which is specific to the client implementation

eta: So you mean user-configurable rules?

Zash, yeah

not rather client-configurable?

I mean this is ultimately flawed anyway because e2ee is a thing

Everything is moot because E2EE

I'm pretty sure there is no place in matrix where you can enter push rule regexes

Is the problem really to be solved on the client-server link? What about some kind of flow control on the s2s side instead? (no idea about the s2s things in xmpp, so maybe that's not doable)

Ge0rG, tada https://matrix.org/docs/spec/client_server/r0.6.1#m-push-rules

Ge0rG: Keywords tho, which might be enough

you can have a "glob-style pattern"

Ugh

eta: that's not what I mean

eta: show me a Riot screenshot where you can define those globs

Ge0rG, hmm, can't you put them into the custom keywords field

If you try to solve it on client side you will invent something like tcp windows. Which is indeed a way to solve ip congestion. And doesn't work here because congestion on the server to client socket doesn't propagate to other links

What was that thing in XEP-0198 that got removed? Wasn't that rate limiting?

Zash: yes

I think the presence-spam-in-large-MUCs issue probably needs some form of lazy loading, right

like, send user presence before they talk

have an API (probably with RSM?) to fetch all user presences

eta: Yeah, that's what I was thinking

the matrix people had pretty much this exact issue and solved it the same way

Oh no, then we need to do it differently!!11!!11!!1 eleven

Zash, it's fine, they use {} brackets and we'll use <> ;P

Phew 😃

the issue with lots of messages in active MUCs is more interesting though

like for me, Conversations chews battery because I'm in like 6-7 really active IRC channels

so my phone never sleeps

I've been thinking I should do some CSI filtering, but then the issue is you fill up the CSI queue

A thing I've almost started stealing from Matrix is room priorities.

So I have a command where I can mark public channels as low-priority, and then nothing from those gets pushed trough CSI

eta: the challenge here indeed is that all messages will bypass CSI, which is not perfect

Zash, yeah, there's that prosody module for that

eta: practically speaking, you might want to have a wordlist that MUC messages must match to be pushed

I almost feel like the ideal solution is something more like

I want the server to join the MUC for me

I don't want my clients to join the MUC (disable autojoin in bookmarks)

and if I get mentioned or something, I want the server to somehow forward the mentioned message

eta: your client still needs to get all the MUC data, eventually

Ge0rG, sure

but, like, I'll get the forwarded message with the highlight

then I can click/tap on the MUC to join it

eta: so CSI with what Zash described is actually good

and then use MAM to lazy-paginate

Ge0rG, yeah, but it fills up in-memory queues serverside

eta: but I think that command is too magic for us mortals

eta: yes, but a hundred messages isn't much in the grand scheme of things

Ge0rG, a hundred is an underestimate ;P

some of the IRC channels have like 100 messages in 5 minutes or something crazy

https://jabber.fu-berlin.de/share/holger/EuIflBOiuR0UyOtA/notifications.jpeg

C'mon guys this is trivial to solve.

my prosody is currently consuming ~ 500kB per online user

https://jabber.fu-berlin.de/share/holger/aIlgwvzEMWv66zF9/notifications.jpeg

Oops.

Zash, also ideally that prosody module would use bookmarks

instead of an ad-hoc command

eta: naah

Bookmarks2 with a priority extension would be cool

we need a per-JID notification preference, like "never" / "always" / "on mention" / "on string match"

which is enforced by the server

Ge0rG: that's a different thing though

eta: is it really?

eta: for mobile devices, CSI-passthrough is only relevant for notification causing messages

Ge0rG: ...actually, yeah, I agree

you want to get pushed all the messages that will trigger a notification

which ironically means that all self-messages get pushed through so that the mobile client can *clear* notifications

which ironically also pushes outgoing Receipts

eta: I'm sure I've written a novel or two on standards regarding that

or maybe just in the prosody issue tracker

eta: also CSI is currently in Last Call, so feel free to add your two cents

Ironically?

Also the topic of notification is just a TODO there.

Heh

> you want to get pushed all the messages that will trigger a notification and that's roughly the same set that you want archived and carbon'd, I think, but not exactly

Ge0rG: wait that sounds like an interesting slide deck

Zash: wild idea, just maintain a MAM archive for "notifications"

I guess a pubsub node would also work

and you shove all said "interesting" messages in there

eta: https://op-co.de/tmp/whats-wrong-with-xmpp-2017.pdf

eta: MAM for the entire stream?

Wait, what's "notifications" here?

Stuff that causes the CSI queue to get flushed? Most of that'll be in MAM already.

Zash: well mentions really

eta: MAM doesn't give you push though

Ge0rG: okay, after reading those slides I'd say that's a pretty good summary and proposal

eta: all it needs is somebody to implement all the moving parts

Break it into smaller (no, even smaller!) pieces and file bug reports?

/correct feature requests*

when I break it into this small pieces, the context gets lost

like just now I realized there might be some smarter way to handle "sent" carbons in CSI, than just passing all through

One huge "do all these things" isn't great either

but maybe a sent carbon of a Receipt isn't too bad after all because it most often comes short after the original message that also pierced CSI?

did I mention that I'm collecting large amounts of data on the number and reason of CSI wakeups?

and that the #1 reason used to be disco#info requests to the client?

Possibly (re carbon-receipts) ✏

Did I mention that I too collected stats on that, until I discovered that storing stats murdered my server?

I'm only "storing" them in prosody.log, and that expires after 14d

but maybe somebody wants to bring them to some use?

disco#info cache helped a *lot* IIRC

I also found that a silly amount of wakeups were due to my own messages on another device, after which I wrote a grace period thing for that.

IIRC before I got rid of stats collection it was mostly client-initiated wakeups that triggered CSI flushes

Zash: "own messages on other device" needs some kind of logic maybe

like: remember the last message direction per JID, only wake up on outgoing read-marker / body when direction changes?

Ge0rG: Consider me, writing here, right now, on my work station. Groupchat messages sent to my phone.

just waking up on outgoing read-marker / body would be a huge improvement already

Zash: yes, that groupchat message is supposed to clear an eventual notification for the groupchat

that = your

After the grace period ends, if there were anything high-priority since the last activity from that other client, then it should push.

Not done that yet tho I thkn

But as long as I'm active at another device, pushing to the phone is of no use

Tricky to handle the case of an incoming message just after typing "brb" and grabbing the phone to leave

Especially with a per-stanza yes/no/maybe function, it'll need a "maybe later" response

Zash: yeah. Everything is HARD

also for all slack's complicated diagrams their notifications don't even work properly either

like it doesn't dismiss them on my phone, etc

Zash> And the way to solve that is to throw data away. Enjoy telling your users that. I'd say that's where there is TCP on top of IP (where I'd argue, the actual congestion and traffic flow control happens)

flow: With TCP, same as XMPP, you just end up filling up buffers and getting OOM'd

Zash, I don't think those two are realy comparable: with tcp you have exactly two endpoints, with xmpp one entity communicates potentially with multiple endpoints (potentially over multiple different s2s links) ✏

Zash, I don't think those two are really comparable: with tcp you have exactly two endpoints, with xmpp one entity communicates potentially with multiple endpoints (potentially over multiple different s2s links) ✏

(me says nothing about mptcp)

So what Ge0rG said about slowing down s2s links?

I did not read the full backlog, could to summarize what Ge0rG said?

(otherwise I have to read it first)

13:31:21 Ge0rG "Holger, Zash: we could implement per-JID s2s backpressure"

besides, aren't in MPTCP still only two endpoints involved (but using potentially multiple paths)? ✏

I am not sure if that is technically possible, the "per-JID" part here alone could be tricky

it appears that implementing backpressure would likely involve signalling back to the sender, but what if the path the sender is also congested?

I'm not sure this is even doable without affecting other users of that s2s link

as of now, the only potential solution I could come up with is keeping the state server side, and have servers notify clients that the state changes, so that clients can sync whenever they want, and especially how fast they want

but that does not solve the problem for servers with poor connectivity

let’s change xmpp-s2s to websockets / http/3 or whatever which supports multiple streams and will of course solve the scheduling issue of streams competing for resources and not at all draw several CVE numbers in that process :)

Not impossible to open more parallell s2s links...

one for each local JID? :) ✏

Heh, you could open a secondary one for big bursts of stanzas like MUC joins and MAM ....

Like I think there were thoughts in the past about using a secondary client connection for vcards

haha wat

Open 2 c2s connections. Use one as normal, presence, chat etc there. except send some requests like for vcards over the other one, since they often contain big binary blobs that then wouldn't block the main connection :)

Well… at this point you may start thinking about removing tcp (its flow control doesn't work in this case anyway) and do something xml over udp instead?

At some point it stopped being XMPP

QUIC solves this