jdev - 2024-08-06

I intend to apply to join the XSF this year and would like a wiki account. Name: Gideon White Username: GideonW Email: gideon@purelymail.com JID: gideon@xmpp.earth Any other info I need to provide for the account?

gideon: ah, good to see new people join! I have created an account for you on the wiki. You should have received an email with a temporary password. That is all you need to create a membership application.

You probably want to add it to https://wiki.xmpp.org/web/Membership_Applications_Q3_2024

One small question, on connect, are you displaying a notification for all the roster items that are in ask="suscribe" ? Or just listing them as the other roster elements ?

Don't display a notification (as in a pop-up one), but do display them somewhere of course

what is the "latest" or "most advanced" stage where a stream reset would occur in XMPP?

is there any case where you would reset a stream after having exchanged a stanza (message/iq/presence)?

I don't think so

Weeeeell... Technically, XEP-0138 doesn't specify when compression is negotiated, so in principle it could be after stanzas have been exchanged. But, XEP-0138 is out of fashion (because of the SeCuRiTy!!!) and even when it wasn't, it was usually done just after authentication (see XEP-0170). I'm assuming we're not counting XEP-0077, though, because that can occur quite early, and before SASL for one thing.

dwd, but, the real reason was the exessive memory usage compression added on servers :)

(one of the reasons it's hard to motivate bringing back compression at least)

Hmm, would it still if servers never kept any per-stream compression state laying around?

You mean, using a stanza framing layer? For me, equally unlikely to be something I do.

Somewhere I might have my notes on compression memory usage, but a naive implementation got something around 5-10% higher compression at a cost of 300-400% of the memory usage. But yeah, compression isn't very interesting in most cases. And when it is interesting, it's often an S2S case.

why not interesting in c2s? that's where all the bandwidth constrained devices are

If anything, I might be interested in compressing pending CSI-held stanzas, that's megabytes for me due to all the MUCs I am in

that could be fun. compression but only with csi

And that would not be externally visible

Just compress data into a buffer, decompress when sending.

or send the compressed buffer and save both sides that pain

Or the inevitable moving of that buffer into persistent storage ... :(

> You mean, using a stanza framing layer? For me, equally unlikely to be something I do. Zash: no I mean the vulnerability was sharing compression context across stanzas, the fix is not to do that, which probably drops memory use ↺

So doing a full flush? I think there were experiments on that, I don't remember what the effects on memory use were, only that compression ratio went to like 50% or something

It still kept some compression context so I am not sure it was a clear win for 50% bytes sent

With zstd and a dictionary tuned for XMPP?

I still need to try it... I imagine it performs pretty well

I don't think zstd existed and IIRC someone pointed out that you can't actually tune a dictionary that way

What do you mean? Zstd explicitly supports training and using a shared dictionary

I don't remember who said it where, but the point was that it didn't work the way we would need it to. Maybe that was wrong. I never got through the Zstd C library to the point of writing a Lua binding anyway, so it was of no use to me. I only remember that it was very complicated.

Okay if I'm reading some words from somewhere¹ correctly, dictionaries work as I think they do, i.e. just like if you were to insert a prefix before your content. So if you then compress a stanza and do a full flush of the state, the dictionary would also be lost and you would start from zero. ¹https://github.com/facebook/zstd/blob/dev/doc/zstd_compression_format.md#dictionary-format

Unless you go and compress each stanza in a fresh state initialized with a dictionary, and then you would need some framing to know where boundaries are. ✏

You already split stanzas on boundaries to route them, same boundaries for clearing compressor state?

Sigh. No.

I was talking about the compressed data, not an XML stream.

Oh, on the recieving end, if needed you can put framing there but not sure if it's needed yet

Anyway, all that's way past the point where I ran out of energy and motivation.

As a quick test on the command line I seperately compressed 3 different text files, cat *.txt.zst > all.zst, and it successfully decompresses to the same as cat *.txt

Which probably means it has a header/framing of some sort built in to the format