XMPP Service Operators - 2021-04-29

WTF Melinda Thompson?

I, too, have questions.

Poor jonas’... Being sued for talking about STUN/TURN security implications on a mailing list

🐈

Maranda: links...links c'mon

Licaon_Kter, https://mail.jabber.org/pipermail/operators/2021-April/003135.html

10x

Nothing says real person more than `"Jane Doe" <evesmith3413@gmail.com>` ¯\_(ツ)_/¯

Relaying e-mail to the police? Sounds like a compromised TURN over SMTP ✏

Anonymus threats is not the same as "beeing sued" That mail is laughable.

very true. I’m not taking this in any way serious, esp. because there’s nothing I’ve done wrong under any jurisdiction affecting me.

could be a translation issue or maybe they didn’t realize that the mail came from a mailing list or maybe google flagged it because DKIM and that made it look suspicious or whatever

Or this is very serious and sooner or later you'll receive an injonction from tribunal not to ask for people to randomize their STUN/TURN ports xD (joke)

What a difference an year makes 2020: OMG https://nitter.fdn.fr/HCornflower/status/1254003453011034112#m & https://nitter.fdn.fr/HCornflower/status/1254068862158614529#m VS 2021: MEH https://mail.jabber.org/pipermail/operators/2021-April/003131.html

lol didn't know the guy so closely follows xmpp 😀

Is simply changing port number really that effective? Even if an enterprising attacker can port scan or lookup SRV records?

I don't have srv records for stun/turn

Wiktor, you are aware that Philipp Hancke is fippo, who has a huge part in the authorship of Jingle?

octagon, scanning all UDP ports + all IPv4 addresses is much more expensive (by a factor of 65535) than just scanning all IPv4 addresses on a well-known port. and yes, the STUN/TURN port is not advertised via SRV typically

jonas’, yeah, but I didn't know he's still interested in that in context of xmpp, I thought it's part of the disconnect of google from xmpp and just using jingle for other purposes (webrtc)

he’s sometimes to be seen on the xeps github as well as standards@

oh, I see

and, apparently, operators@

always criticizing... I get it 🙂

and Licaon_Kter, I don’t blame him for questioning this... I am also fairly certain that the email has to be read as "are you sure this is being attacked? if so... we need to ask critically why, because it does no seem to be the amplification factor alone which makes this interesting".

Wiktor, I’d call it "asking the hard questions nobody wants to think about" especially in the linked context

Ofcourse, it just pinged when I saw his name and remembered those remarks on Conversations A/V launch.

Yes, I want to see the links to the attacks too...as I don't see any random stun/turn connections here. But when I hosted SIP I had >1000 connections on 5060/5061, no visible impact but they were there in my router ✏

Licaon_Kter, how are you measuring your STUN/TURN "connections"?

I see them in ejabberd logs, but was expecting activity on the router/firewall

I'm not measuring, actively, but RPi's are rather sensible to load :)

Licaon_Kter, maybe do a tcpdump on udp port 3478

What a mess nowadays to allow two peers to exchange data... STUN/TURNS/other protocols, server ressources... all because of ipv4 is still alive. Internet failure.

All because ipv6 was just born we should do what? pepta.net

Us: nothing. But ICANN could have speed up things mandating every ISP to provide IPv6 everywhere and deprecate (and kill) ipv4 with deadlines. ✏

Dual stack seems more now forever than a temporary transitional situation. Making everyone to suffer from ipv4 cons.

BWAHAHAHA

Except we can provide IPv6 compatible services whenever possible. To make things evolve step-by-step. ✏

Who's this we? How can I provide anything where there's no 6 here?

We = service providers. If no 6 available, can't ask for the impossible. Maybe consider ISP/hosting switching, or any action to manifest your disappointment to your providers.

No 6 in sight... But see, they announced 6 back in '15 so PR side all is well

Couldn't you filter the IPs you accept connections to your stun server on based on the IPs of clients currently connected to your XMPP server?

moparisthebest, no, you also need to accept packets from their peers

AIUI

at least once the channel is established

though possibly that is only after the initial STUN phase

and the initial STUN phase is the problme

moparisthebest, but with CGNATs, you can never know whether they always end up with the same egress IP for the same flow

and CGNATs is where you need it most :(

So we need our own stun protocol replacement with an auth secret ? :/

and of course it requires a live channel between your xmppd and the stun server… which people wanted to avoid with REST mode

moparisthebest, I don’t even know at this point what the attackers want to achieve

so I refrain from making improvement suggestions at this ponit

Or between your xmppd and the firewall in front of your stun server

same thing really :)

also I’m not sure how well that scales…

So if you accidentally blocked stun for unfortunate cgnat people, they'd just use turn right?

no, you need STUN before you TURN AFAIK

If so, I think they'd have to do that anyway

I need to dig deeper into ICE

however, at this point, moving ports has mitigated the traffic 100%, so I’m not going to worry about that until the abuse alert triggers on the new port ;)

It tries stun before turn before, the question is if no stun response will it go to turn

Moving ports is at most a temporary fix, scanning the entire ipv4 internet is easy

scanning IPv4 + all UDP ports is not that easy

Likely shodan would already tell you about all stun ports

if you need one day to scan the entire IPv4 range, you need 2^16 days for that on all UDP ports.

even if we’re going to assume that only half of the port range is used and that you only need to scan every other port because of the alt port mechanic, that’s still 16kdays

I guess it'd be easy to test what no stun, only turn, looks like in practice, just turn it off/block the port and see if calls go through

https://github.com/robertdavidgraham/masscan can scan the entire internet in <5 minutes so I wouldn't count on numbers saving you here

moparisthebest, jonas', yes only the initial phase is the problem, a peer port is only allocated after successful TURN authentication (and can't be used for such an attack then either).

And yes you could avoid the issue by having some integration with the XMPP server.

as far as scaling goes, ipset https://wiki.archlinux.org/index.php/Ipset can easily whitelist more IPs than your XMPP server can handle, doing this on not-linux is left as an exercise for the user

ipset is dead tho? but I suppose nft has the same mechanism.

Dual stack foo will be fun again though.

> https://github.com/robertdavidgraham/masscan can scan the entire internet in <5 minutes so I wouldn't count on numbers saving you here Read full README before trying ;-) > Scanning the entire Internet is bad. For one thing, parts of the Internet react badly to being scanned. For another thing, some sites track scans and add you to a ban list, which will get you firewalled from useful parts of the Internet.

Yes that's what I mean (or vice-versa).

moparisthebest, oh yes. challenge: a client is connected via IPv6, but its peer is connected via IPv4. Thus ICE can only succeed via v4, meaning that the STUN server needs to be reached over v4, but the v4 address of the client is not known by the xmppd and thus not allowlisted in the firewall. ✏

and in theory, it can't reach STUN and falls back to TURN, no problem

moparisthebest, but then you’re using TURN which sucks.

for no good reason, too

the good reason is avoiding STUN amplification attacks I guess

It may well try to reach TURN via v4 and v6.

You don't want one of those to fail.

you don't need to ip-whitelist the TURN ports

Huh?

There's no such thing as a TURN-only port.

Protocol-wise, TURN is a STUN extension.

then you can't do this with iptables/firewall (unless it can tell the difference between STUN and TURN) and it needs done in the STUN/TURN server itself ?

so that you can ip-whitelist STUN but allow-all with TURN ?

I understand nothing.

The entire idea only works if you handle STUN and TURN the same way. Regardless of whether you involce the STUN/TURN server or just the firewall.

moparisthebest: where do you get the ip's to whitelist exactly? ✏

Licaon_Kter, the client IPs currently connected to your XMPP server

Right. I would think this could work (for STUN and TURN), modulo dual-stack foo (as usual).

Holger, pseudo-code, in the turn server accept packet bit: if packet.is_stun() && packet.ip_is_whitelisted() { packet.respond(); } else if packet.is_turn() { packet.respond(); } // otherwise no response to packet

moparisthebest: As I said, that's not how it works.

since stun isn't authenticated, you only respond if the IP is whitelisted, turn is authenticated so no need to whitelist there

but it *can* work that way, no ?

That's not how it works.

No.

why not ?

Only by changing the protocol.

If it worked that way, you could just enforce authentication for STUN as well.

It doesn't work because the attack happens in the pre-auth phase.

And authentication is challenge-response.

No way to avoid sending the challenge to the spoofed address.

so the server talks first even in the auth/turn scenario? :/

And at this point there's no difference between STUN and TURN.

The server sends an "auth required, here's your challenge" in the auth/TURN scenario, yes.

The server *responds* with that "auth required, here's your challenge" packet to the initial client query.

But it's not obvious to me anyway how the idea depends on handling TURN in different ways.

ugh, so yea, I guess it's: 1. fix STUN/TURN protocol 2. XEP so client sends all IPs it might use to the server after login for whitelisting :'(

2. alone would do the trick I guess.

if the initial TURN packet from the client was "hi start a TURN session and this is my authentication key: stnhaoeunsth" that could be let through without whitelisting

but, apparently that's not how it works

moparisthebest: AFAIK you cannot rely on ip src to be the same for two separate connexions because of CGNAT.

Right, if we had a way to negotiate that key outside of TURN.

yep it's either 1 or 2, but 2 is the only thing I think the XMPP community alone can reasonable do ?

pepta.net, yep jonas’ said as much, I don't have a solution there

Indeed.

what about... kinda port knocking ?

you get a secret from the XMPP server, send it to the STUN port, and that IP is whitelisted for the real STUN request that comes later ?

you can’t do (2) though

a client doesn’t know it’s IP … it needs STUN for that ;D

right, so you have a pre-STUN reverse-STUN that whitelists the client's IP for the real STUN ? :P

Ah indeed :-)

"well"

https://xmpp.org/extensions/inbox/sic.html needed ?

moparisthebest, as I also said on-list… there are much more lucrative amplification attack targets than STUN servers out there.

It makes absolutely no sense to abuse stun for that

I don't think expecting sense from attackers will work very well

oh I think it does

and I think we must

otherwise we might be fixing the wrong thing

if we fix it so our STUN servers can't be used for amplification at all, it doesn't matter what the attackers do ?

no

that’s not true

ITYM can’t be used for reflection, in which case it might indeed be true

because not all amplification fixes (such as requiring a minimum inbound payload of 100 bytes, e.g. padding) may fix other aspects of the attack

I was just talking about adding authentication, so an authenticated XMPP client can do what they want

we have that in theory… for TURN anyway (and leaving out the fact that the not authorized message is, again, larger than the request)

We don't.

I mean.

we don’t?

The point would be accepting an auth token/password rather than doing challenge/response.

We do have 0215 to communicate a token to the client.

isn’t that how it’s used?

But STUN/TURN doesn't have a way of checking that in the initial request.

do I need to read the TURN RFC?

so my hack proposal was a new auth'd protocol to whitelist the client's IP, and then proceed with regular STUN/TURN

I keep repeating myself on this point as well but people keep ignoring me until Philipp Hanck pops up 😂️

I start regretting bringing this up

> he server sends an "auth required, here's your challenge" in the auth/TURN scenario, yes.

moparisthebest: > the client IPs currently connected to your XMPP server Wait...but... > you also need to accept packets from their peers ...

I mean literally something that listens for a single UDP packet with "auth my IP please, auth code: XXXX", and it only responds with "ok" if the auth code is correct

then regular STUN/TURN takes over

Because back then people like Neustradamus explained how PLAIN auth is the end of the world and all we can allow is MD5-based challenge-response.

So now all we have to check a temporary token is MD5-based challenge-response.

moparisthebest, and you'd add that to what spec?

Can't we just rate limit the port with iptables?

Holger, I guess a XEP specifies sending the auth code, and maybe the UDP packet ? :/

octagon, would make it easy DOS?

octagon, rate limiting, even by IP subnet, was surprisingly tricky. Tried that.

not tricky to set up, but tricky to get right

call attempts cause packet bursts, and if you have to retry within a given time frame because of an unrelated error, this will make it harder for you to succeed at having a call

might be better if you also filter for specific STUN packet types, not sure if I went that far before moving on to changing the port

moparisthebest, currently, my hypothesis is that we’re already golden if we change ports because reasons.

even if the stun servers are rediscovered, they may not be useful for the type of attack

changing ports is good, causes no harm, we should do it, I'm just uncomfortable with that being the final solution

moparisthebest, I would do that and reconsider more drastic measures when the attacks come back

Actually!!

and you can randomize ports within a time-limited window, to enhance the proposed work-around.

takes too long to actually do something else though, I'd rather start now

croax, you can indeed

I totally forgot about the Short-Term Credential Mechanism (which is completely unused AFAIK): https://tools.ietf.org/html/rfc5389#section-10.1

But possibly precisely what we'd need.

moparisthebest, yep if we have the time we should do the proper fix and go ahead and fix STUN/TURN themselves where necessary

I.e. my "all we have is challenge-response" claims were incorrect 🙂

Holger, ehh, I was always assuming that that’s exactly what clients do?!

No they always do long-term.

that does seem exactly what is needed Holger , what are the chances libwebrtc has support :/

Not sure e.g. libwebrtc will even implement short-term.

moparisthebest, right, that's the question.

well, libwebrtc is on github, isn’t it?

shouldn’t be that hard to implement support?

add it to libnice and be golden

it's google's

worth trying I guess

Either way maybe that's the proper solution.

if it was any other company, I'd be more optimistic

And then add to 0215 that we do short-term (unlike the rest of the world).

Well, good enough to recommend that.

Does Coturn support it?

(eturnal doesn't ...)

I am happy to implement it there if it helps

should be just a bit of C++ I guess

Just curious.

"should be just a bit of C++" famous last words 🤣️

Holger, it caims to support 5389

Well I claim that as well 😂️

But yes may well be supported of course.

``` -a, --lt-cred-mech Use long-term credentials mechanism (this one you need for WebRTC usage). ```

so I guess the default is short term?

I don’t set that though

Would sound weird to me but either way "WebRTC usage" is exactly what we're doing.

``` 1/24/2015 Oleg Moskalenko <mom040267@gmail.com> Version 4.4.1.1 'Ardee West': - https admin server; - SSLv2 support cancelled (security concern fixed); - The server-side short-term credentials mechanism support cancelled; ```

"oh no"

So maybe it's a simple `git revert` 🙂

Revert revert!

``` both modified: ChangeLog both modified: INSTALL both modified: README.turnserver both modified: examples/etc/turnserver.conf both modified: examples/var/db/turndb both modified: man/man1/turnserver.1 both modified: rpm/build.settings.sh both modified: rpm/turnserver.spec both modified: src/apps/relay/dbdrivers/dbd_mongo.c both modified: src/apps/relay/dbdrivers/dbd_mysql.c both modified: src/apps/relay/dbdrivers/dbd_pgsql.c both modified: src/apps/relay/dbdrivers/dbd_redis.c both modified: src/apps/relay/dbdrivers/dbd_sqlite.c both modified: src/apps/relay/dbdrivers/dbdriver.h both modified: src/apps/relay/mainrelay.c both modified: src/apps/relay/userdb.c both modified: src/apps/relay/userdb.h both modified: src/apps/uclient/mainuclient.c both modified: src/client/ns_turn_msg.h both modified: src/ns_turn_defs.h both modified: src/server/ns_turn_server.c both modified: turndb/schema.mongo.sh both modified: turndb/testmongosetup.sh ```

A simple `git revert` with a tiny bit of merge work.

I wonder if that means they won't accept it upstream though

So I have a new secret plan how to sell my eturnal thingy.

Short-term-cred version will be expensive.

Inb4 Holger hired by Google

One question is whether libraries such as libwebrtc support short-term creds, the other is whether they support authenticated STUN.

you'd still need to violate that RFC a bit

o If the message does not contain both a MESSAGE-INTEGRITY and a USERNAME attribute: * If the message is a request, the server MUST reject the request with an error response. This response MUST use an error code of 400 (Bad Request).

you'd need to just not respond

same with invalid username/password

Yes the TURN server would have to be configurable to work that way.

or if the response was smaller it'd be ok but no idea if that's the case here

I still doubt it's about the response size 🙂

yea you are right, it's less a problem if response is less, but a problem anyway

As long as factor is <= 1 it would no more be a problem as attacker are supposed to only exploit this to the benefit of data amplification. ✏

croax, it stops data amplification, but still allows them to hide their true source IP

Yet, their are Windows user for that. :-) ✏

But agree with you.

Any UDP service without authentication would be vulnerable the same. Eg NTP, DNS, ... ? ✏

yep

any that responds without authentication anyway, for example wireguard is not susceptible

croax, NTP and DNS are prime examples of such offenders, yes

moparisthebest, do you know how QUIC does things?

jonas’: https://tools.ietf.org/html/draft-ietf-quic-transport-34#section-8.1

So QUIC seems to think 3x amplification is ok...

You should run it on a different port :)

That is finally supported, though you can just ask DNS where that is