XMPP Service Operators - 2021-05-14

tom: as moparisthebest said, you can use the new statistics features in prosody trunk to gather nice stanza data

I am also working on stuff which bypasses the lua GC during xml parsing, let's see how that goes

prosody-trunk will also keep counters on stream closure reasons

so you’ll see when a stream gets killed with policy-violation

for instance: host=search.jabber.network direction=outgoing error=connection-timeout 59.6 μevents/s [avg] host=search.jabber.network direction=outgoing error=:remote-choice 1.92 mevents/s [avg] host=search.jabber.network direction=outgoing error=not-well-formed 118 μevents/s [avg] host=search.jabber.network direction=outgoing error=unsupported-stanza-type 3.1 mevents/s [avg] host=search.jabber.network direction=outgoing error=:local-choice 69.3 mevents/s [avg] host=search.jabber.network direction=incoming error=:local-choice 64.5 mevents/s [avg] host=search.jabber.network direction=incoming error=:remote-choice 6.32 mevents/s [avg]

Bypassing GC by... Pooling? FFI into Rust? Enquiring minds want to know! :)

» <jonas’> tom: as moparisthebest said, you can use the new statistics features in prosody trunk to gather nice stanza data thankyou

By the way

I just switched my rootfs from ext4 to zfs

It nearly halved my storage utilization in half

44G used before, not only 26.7G

Lz4 compression alone default settings

FWIW the 10MB limit was originally added to defend against zip bombs, and also an in-the-wild attack from a script that simply opened an attribute and sent an infinite number of bytes

That script was used to attempt to DoS a popular service based on XMPP, demanding payment to stop the attack

(I believe the service was running jabberd2 at the time)

There has to be some limit. If it's 17MB, fine. But then at a minimum that means someone can open a connection to your server and cause 17MB RAM usage. In reality it's more, because the data is parsed and there are other related overheads

But if that's okay with you, fine. But combined with the GC issue, it means that memory is never/rarely released

My plan/hope is that I/someone will write a XEP and a module that simply advertises stream limits so that an entity can preemptively bounce a stanza instead of DoSing the whole XMPP stream

👍

FWIW I've just set up a new Nextcloud instance the other day and had to bump several Nginx and PHP limits in order to be able to upload pics larger than 1 MiB. And those are just about transferring binary blobs, not structured XML data to be parsed by the server.

> FWIW the 10MB limit was originally added to defend against zip bombs how does that logic work out, cause a zip bomb is <1MB

qrpnxz, XMPP streams typically used to be run with a compression layer enabled. A malicious client would send, a 10 byte packet that expanded into a 10GB stanza, for example

Instant DoS

We didn't want to break stuff in a security release (we wanted people to upgrade to fix this issue, without the fear of normal stuff breaking), so we chose 10MB as a reasonable limit, with the intention of later choosing some more appropriate limits in a future major release

But we never settled on what appropriate limits are. Now we find that 10MB is problematic even without compression for most servers, so here we are :)

> qrpnxz, XMPP streams typically used to be run with a compression layer enabled. A malicious client would send, a 10 byte packet that expanded into a 10GB stanza, for example Can't unpacking be stopped if unpacked data reaches a certain size?

Yes, that's exactly what the 10MB did

Combined with an API we had to add to lua-zlib to allow us to say "no more data please"

MattJ> qrpnxz, XMPP streams typically used to be run with a compression layer enabled. what?

ohh

I maneged to miss the `used` in that sentence thrice.

It surprised me to encounter an error 403 on a public jabber server, which means it requires a "Referrer Header" to be sent by my browser in order to log in. I am a little confused now since Firefox just recently has made it possible to turn off the option of sending Referrer Header" as it was raising some privacy concerns. Now this requirement from the server is justified as by the error message "for security reasons, to ensure that your browser is not being hijacked by a third parties". It is not clear to me why suddenly Firefox gives us an option to disable this "important security option" to protect our privacy, quoting here the title from their blog "Firefox 87 trims HTTP Referrers by default to protect user privacy", but on the other hand expose us to carnivorous internet browser hijackers? What is the game here and what is the real threat?

the message from that jabber server is misleading

what it wants to say is "We use Referrer to validate that you previously were on our page and that this isn’t some script running an CSRF attack aganist us"

however, referrer is just an means to an end -- disablign it doesn’t make *you* more or less vulnerable

it just prevents this specific server-side mitigation (and to be fair, referrer is a pretty terrible way of doing this)

(because referrer is privacy invasive)

the industry has other ways of dealing with CSRF, so don’t worry. firefox is doing the right thing and that serevr operator needs to update their defenses.

jonas’: Thanks for making this clear. Referrer is in my opinion nothing else but a tracking method. When disclosing our private data is justified as "for our own security", it makes my senses tingling.

Ivan, the server-side check is probably from a different time; not actively malicious.

Ivan: I use a "smart referer" plugin that only sends it for pages on the same domain, which allows old checks like that to pass without privacy issues

moparisthebest: thanks, will give it a look

Looking at my im observatory report, how do you enable dnssec on srv records?

DNSSEC is enabled on an entire domain's records or not

Ah, ya makes sense. I haven't done that for the new vhost