XMPP Service Operators - 2021-04-30

moparisthebest, didn't see SRV in the linked draft so how is the port discovery being done in QUIC?

how is port discovery done in TCP?

moparisthebest, are you confuslating QUIC and HTTP/3 there?

nmap?

christian, no offense, but I’d really appreciate it if you could read more than the last two lines of backlog before replying. Thanks! :)

jonas’: today are only two.

christian, maybe use a client with MUC-MAM support then

or check the logs online at https://logs.xmpp.org/operators/ if that’s not an option

re-winding the conversation context of the past day is exhausting ✏

jonas’: okay. Sorry for disturbing.

christian, thanks!

Are there actually ddos attacks using STUN services as a reflector or only the theoretical possibility? I've never seen one an there are way "better" opportunities.

kahlb: there are actual attacks

Are there any resources about it? Analysis? I'm really interested.

there are some script kiddies who discovered "ping -f" and some security guys who feel important :)))

kahlb, tcpdump -enni any port 3478 ;)

jonas’, the theoretical possibility is out of question. But is there a real danger, like booter services actually doing this? I see no reason why they should as long as there are so many highly efficient reflectors out there.

on my system there is no suspicios traffic. Exploiting a reflectro always requires a scan in advance, so maybe i'm safe. But in ddos traffic from larger attacks I've seen lately I never saw anything stun related

Is that what these are: » Apr 30 00:45:57 web turnserver: 974814: session 001000000000133270: realm <> user <>: incoming packet BINDING processed, success » Apr 30 00:45:57 web turnserver: 974814: session 001000000000133270: realm <> user <>: incoming packet BINDING processed, success » Apr 30 00:45:57 web turnserver: 974814: session 001000000000133270: realm <> user <>: incoming packet BINDING processed, success » Apr 30 00:45:57 web turnserver: 974814: session 001000000000133270: realm <> user <>: incoming packet BINDING processed, success » Apr 30 00:45:57 web turnserver: 974814: session 001000000000133270: realm <> user <>: incoming packet BINDING processed, success » Apr 30 00:45:57 web turnserver: 974814: session 001000000000133270: realm <> user <>: incoming packet BINDING processed, success » Apr 30 00:45:57 web turnserver: 974814: session 001000000000133270: realm <> user <>: incoming packet BINDING processed, success » Apr 30 00:45:57 web turnserver: 974814: session 001000000000133270: realm <> user <>: incoming packet BINDING processed, success » Apr 30 00:45:57 web turnserver: 974814: session 001000000000133270: realm <> user <>: incoming packet BINDING processed, success » Apr 30 00:45:57 web turnserver: 974814: session 001000000000133270: realm <> user <>: incoming packet BINDING processed, success » Apr 30 00:45:57 web turnserver: 974814: session 001000000000133270: realm <> user <>: incoming packet BINDING processed, success » Apr 30 00:45:58 web turnserver: 974814: session 001000000000133270: realm <> user <>: incoming packet BINDING processed, success » Apr 30 00:45:58 web turnserver: 974814: session 001000000000133270: realm <> user <>: incoming packet BINDING processed, success » Apr 30 00:45:58 web turnserver: 974814: session 001000000000133270: realm <> user <>: incoming packet BINDING processed, success » Apr 30 00:45:58 web turnserver: 974814: session 001000000000133270: realm <> user <>: incoming packet BINDING processed, success » Apr 30 00:45:58 web turnserver: 974814: session 001000000000133270: realm <> user <>: incoming packet BINDING processed, success » Apr 30 00:45:58 web turnserver: 974815: session 001000000000133270: realm <> user <>: incoming packet BINDING processed, success » Apr 30 00:45:58 web turnserver: 974815: session 001000000000133270: realm <> user <>: incoming packet BINDING processed, success » Apr 30 00:45:58 web turnserver: 974815: session 001000000000133270: realm <> user <>: incoming packet BINDING processed, success » Apr 30 00:45:58 web turnserver: 974815: session 001000000000133270: realm <> user <>: incoming packet BINDING processed, success » Apr 30 00:45:58 web turnserver: 974815: session 001000000000133270: realm <> user <>: incoming packet BINDING processed, success » Apr 30 00:45:58 web turnserver: 974815: session 001000000000133270: realm <> user <>: incoming packet BINDING processed, success » Apr 30 00:45:58 web turnserver: 974815: session 001000000000133270: realm <> user <>: incoming packet BINDING processed, success » Apr 30 00:45:58 web turnserver: 974815: session 001000000000133270: realm <> user <>: incoming packet BINDING processed, success » Apr 30 00:45:58 web turnserver: 974815: session 001000000000133270: realm <> user <>: incoming packet BINDING processed, success » Apr 30 00:45:58 web turnserver: 974815: session 001000000000133270: realm <> user <>: incoming packet BINDING processed, success » Apr 30 00:45:58 web turnserver: 974815: session 001000000000133270: realm <> user <>: incoming packet BINDING processed, success » Apr 30 00:45:58 web turnserver: 974815: session 001000000000133270: realm <> user <>: incoming packet BINDING processed, success » Apr 30 00:45:58 web turnserver: 974815: session 001000000000133270: realm <> user <>: incoming packet BINDING processed, success » Apr 30 00:45:58 web turnserver: 974815: session 001000000000133270: realm <> user <>: incoming packet BINDING processed, success » Apr 30 00:45:58 web turnserver: 974815: session 001000000000133270: realm <> user <>: incoming packet BINDING processed, success » Apr 30 00:45:58 web turnserver: 974815: session 001000000000133270: realm <> user <>: incoming packet BINDING processed, success » Apr 30 00:45:58 web turnserver: 974815: session 001000000000133270: realm <> user <>: incoming packet BINDING processed, success » Apr 30 00:45:59 web turnserver: 974815: session 001000000000133270: realm <> user <>: incoming packet BINDING processed, success » Apr 30 00:45:59 web turnserver: 974815: session 001000000000133270: realm <> user <>: incoming packet BINDING processed, success » Apr 30 00:45:59 web turnserver: 974815: session 001000000000133270: realm <> user <>: incoming packet BINDING processed, success » Apr 30 00:45:59 web turnserver: 974815: session 001000000000133270: realm <> user <>: incoming packet BINDING processed, success » (END) »

It's creating like a gigabyte of logs per day

I am no logging at all stun.

jonas’, Okay, but that's rather ddosing yourself rather than anybody else. What is the target of the attack?

s/ddosing/dosing

tom, yess, that looks relevant

kahlb, we don’t know what the goal is.

just that it is happening in practice

Is there a way to actually filter this

Because i'm not going to do security by obscurity by changing port numbers

If there's some way to definitively tell what is attack traffic and what's not

Either an nftables rule or some kind of logging+fail2ban I'm all for that and will implement fixes

Otherwise I don't really care

Who is being ddosed anyways?

jonas’, the target of a reflector attack is alway the (spoofed) source ip, so you shoud see it in your tcpdump. If it's alway the same, just filter it.

Techniclly you can reflection attack someone with TCP RST

What am I to do, stop sending TCP resets?

kahlb, been there it’s not that easy

Btw, I'm showing 52.113.61.170 being attacked

Which is Microsoft Corporation

Like Azure hosting?

I'm not really in a rush to do any favors to a corporation who I spent the last 2/3 decades protesting the war against open standards and protocols

*their

Or a corporation who patent trolls

ralphm, I don’t know if azure is a separate AS, the whois of that IP only points at MSFT :/

I also wish I knew

Or is a member of the PRISM mass surveillance program

You all remember skype? Or internet explorer

tcpdump says continously :( ``` 07:55:21.376197 In MAC ethertype IPv4 (0x0800), length 64: SOMEIP.3479 > MYLANIP.3478: UDP, length 20 07:55:21.377734 Out MAC ethertype IPv4 (0x0800), length 96: MYLANIP.3478 > SOMEIP.3479: UDP, length 52 ``` ...have I got the disease?

Or outlook blocking your mail server

Licaon_Kter, likely

though, that looks more fun

tom: saw the same IP too iirc

because it’s source port 3479

52.113.61.170 is MS Azure.

Now it's 137.74.0.162

kahlb, how do you know?

Licaon_Kter, hm, OVH, haven’t seen that yet.

Some reverse ip online tools yielded nothing, but one mentined ns1.azure though

You can download the IP ranges Azure uses

Now it stopped...

137.74.0.162 is a TS3 Server. Actually a potential target for DDoS

.162 used port 9987

And...it's back .162:9987

kahlb: so it may be part of a botnet

I've changed the STUN/TURN port to a random one two weeks ago, and I'm still receiving ingress traffic on 3478

20 in 52 out means >2x

Yeah just got that change to

137.74.0.162

It's weird that the sourceport is always the same

9987

They are hitting my server with 9987 as well

ralphm, I don’t think that the .162 is the true source

Looks like I'll change the port then, daaaammit

No it's not that just the attack target

Licaon_Kter, tcpdump does not count IP and UDP headers, you have to add 44 bytes to that IIRC. ✏

Ge0rG, because the attackers only scan once or every few months. They don't get an idea if your IP is still a usable reflector, only if they scan again.

This happens because ISPs don't implement BCP38, so you can spoof the ip address of anybody

Licaon_Kter, tcpdump does not count IP and UDP headers, you have to add ~44 bytes~ 28 bytes(?) ✏

kahlb, we changed ports >2w back.

IIRC.

redacted.3478 > 137.74.0.162.9987: [bad udp cksum 0x979f -> 0x4ff1!] UDP, length 108

Hey

Half these packets have bad checksums too

tom, that’s not real ✏

Whoever the ddoser is they aren't very skilled

Here is some really good literature for those who want to dig in: https://christian-rossow.de/publications/amplification-ndss2014.pdf

that is the packet your server sends, and depending on offloading settings, checksums are not correct (yet) when tcpdump sees tem

Bad checksums and using the same source port continuously

so you are "sending" a bad checksum in that line, not them

Rossow has a few more publications about it, all are worth reading.

The only problem is, there's a bug in linux's nft where it blocks some needed ipv6 multicast if you enable bogon filtering

Oh my bad your right

That's my packet

Hey it stopped

Did it stop for anyone else as well ?

Isn't dnssec making the spoofing harder?

christian, no, dns has nothing to do with this.

The problem is ISPs are too lazy to implement https://tools.ietf.org/html/bcp38 on their core router

Which makes ip spoofing impossible

Oh, they are back

> Did it stop for anyone else as well ? It's on and off Changed ports so it's quiet for now 215 to the rescue ✏

Here's an idea

Log every binding request along with a timestamp and auto-ban ips that do over 120 binding requests /minute

I can't think if a scenario where there would be that many binding requests from the same ip per minute

*of

What do you think?

Now if i can see how to configure coturn to log the info

`verbose`

tom: this approach has a problem where the attackers will still see your server as a reflector on their scans, and then still abuse you.

Sure some packets will sneak through, but they can't use your server for very long

At least

They'd have to keep changing source ips

tom, 120 / minute is hard to measure.

Fail2ban could do that

that’s 2/second

Which they are exceeding

I'm showing ~15/second

problem with 2/second is, a legitimate call can produce that

also, gentle reminder that this is a public place and the attacker(s) may be listening in

be careful which countermeasures and details you share publicly

» problem with 2/second is, a legitimate call can produce that 2 binding requests per second for 1 minute?

That's why i said per minute not per second

no, not for one minute. just saying that measuring this correctly is fun

Ok

I just want to say that coturn's configuration file options and syntax is terrible

Wish there was a better turn server I could use

eturnal? :)

also, run your coturn with --prod to shave 20 bytes off the response :)

I'll check that out

The best thing would be: identify the scanner IP and blacklist it. Additionally take some of the countermeasures you just mentioned. As long as the scanner IP doesn't change it'd stop considering your system as vulnerable and the attacks would stop. But it's difficult to find these IPs. Best: Find systems *without* stun, search their firewall logs for UDP attempts on port 3478 and look wher they're coming from, since that can't be legit traffic. Hopefully there aren't so many, when in doubt just block em all. Maybe we should consider a public blacklist, though that could lead to countermeasures from attackers.

kahlb: idk

Trying to stop scanning on the internet seems like a fool's errand

you can't stop scanning. But you can stop your system being identified as a reflector. At least up to some point.

» --secure-stun » Require authentication of the STUN Binding request. By default, the clients are allowed anonymous access to the STUN Binding functionality. »

Makes it worse as the reply gets larger afaik.

When media session between 2 contacts on different servers, does each contact use its own STUN/TURN or only caller one?

Ge0rG: jonas’ you should definitely turn on secure-turn first

That way, all your legit users of your turn server will be differentiated from ddos spoof users

Then it will be much easier to instead of block on n binding requests per minute, block on n 401 UNAUTHORIZEDs per minute

croax, both might or might not be used. It doesn't depend on who is the caller.

However, coturn has an absolute garbage logging syntax where things will need to be matched on multiple lines

tom, I'm slightly biased but do believe eturnal's logging is way better.

And config 🙂

Thanks. I will be checking that out, I just can't change turn/stun servers today

tom, main downside I see is, no official distro packages (yet).

croax, basically if your client isn't reachable directly by your peer, your TURN server is used.

croax, so if neither your nor your peer's client is directly reachable, both TURN servers are used.

Holger: thanks for clarification.

tom, except that this is about STUN, not TURN

jonas’: still applicable

tom, question is: is the Not Authorized reply even larger? (IIRC the answer is "yes")

and as I understand the curent state of STUN/TURN/WebRTC affairs (and as we discussed yesterday), (legitimate) clients *have to* first fail authentication before they can authenticate because of challenge/response nonsense

Yes.

First auth step is to receive an UNAUTHORIZED response from the server.

tom: > --secure-stun Do you use that? I would've been unsure whether e.g. libwebrtc copes with that.

It does

Nextcloud works with that too

Interesting. (I wonder what the point is though.)

more traffic, of course

Do you wanna test it yourself?

Send some p2p file transfer with jingle

I asked a random guy on the Internet specifically because I've always been too lazy to test myself :-P

Because I didn't see any point anyway. Until we had that short-term-auth idea yesterday: If libwebrtc supports short-term _and_ has a code path for authenticated STUN, maybe things would just work. Maybe.

and we get the STUN servers to STFU on authentication failure.

or maybe it doesn’t matter anymore then because the proper request packet is already large enough

I'd be way more afraid of submitting libwebrtc patches than of patching STUN servers.

worth a shot I guess

Good

except that it needs someone to write the code

I mean the Coturn guy is quite responsive, but who knows how easy it is to convince Google to merge some functionality eveyone else considers outdated.

Holger, did you already talk to coturn about short-term-auth?

I mean worst case we have to go through IETF to convince folks

jonas': No I didn't talk to him but I mean you could look at the old code for the short-term-auth support, and I guess an option to drop other traffic is trivial.

question is why it was removed

if they would be interested in readding it

As I said nobody uses it.

is that why it was dropped or is that your guess why it was dropped?

But Coturn is a bloated monster, I doubt he'll strongly oppose re-adding it if you use it.

My guess.

:)

Why else would it be dropped.

undisclosed security issue; impossibility to integrate with the remaining architecture; ...

I could add it to eturnal, you post an issue that Coturn is missing that feature in comparison and he'll add the code himself :-)

Nah.

And I don't think you need the IETF for a `drop-other-traffic` option.

but for a PoC we can easily go ahead with just eturnal

Yes. I'm just trying to explain why I think the real question is whether libwebrtc supports it.

https://webrtc.googlesource.com/src/

I'll have a look

-EATWORK, otherwise I’d look too

I keep re-cloning that into /tmp once every three months. Because this is the very last time I'm interested in it ...

I think you need to accept your fate that you’re now a webrtc person ;)

Doesn't look like libwebrtc supports short-term.

Ah yes

Goolag Korporation

Holger, any clue how a client using libwebrtc would indicate to it that it is supposed to use short term instead of long term?

could libwebrtc just *try* short term and then fall back to long term based on the response to short term, or does that not work in stun?

And now I remember that the TURN RFC has strong language towards long-term auth. (I had a discussion on whether to interpret it as long-term being mandatory last year or so. Why didn't I remember any of that yesterday? My brain is useless.)

The idea as per the STUN RFC is that client and server know what auth mechanism to expect before-hand. I.e. I'd imagine the libwebrtc user (Conversations) to specify a "do short-term" flag.

You obviously can't signal the desired mechanism without adding a round-trip, which is precisely the thing we'd like to avoid.

From the TURN RFC: ``` The server MUST require that the request be authenticated. This authentication MUST be done using the long-term credential mechanism of [RFC5389] unless the client and server agree to use another mechanism through some procedure outside the scope of this document. ```

So there *is* a bit of room for an out-of-scope procedure :-)

``` [RFC5389] specifies an authentication mechanism called the long-term credential mechanism. TURN servers and clients MUST implement this mechanism. The server MUST demand that all requests from the client be authenticated using this mechanism, or that a equally strong or stronger mechanism for client authentication is used. ```

Those sentences won't make it easier to convince Google I guess ...

Maybe just change ports, call it a day, and spend your time on MIX :-P

I'm not _that_ concerned about the actual problem, but it _is_ a little annoying how the specs basically have exactly the thing that would make sense for our use case. And generally for the WebRTC use case.

yeah

annoying

> Why didn't I remember any of that yesterday? page faults take a while to service also in human brains ;)

That RFC text is odd no? "You MUST do X, unless you don't want to"

Sounds like a SHOULD

There _must_ be an auth mechanism. Long-term or equivalent/stronger. But no weaker/none. _Should_ would allow _none_ AFAIK

The first of my above two quotes doesn't contain the "stronger" part though.

Which makes no sense for a protocol spec anyway.

And saying "client and server can agree on breaking the spec in arbitrary ways" is stating the obvious.

So effectively this does work like a proper MUST. A TURN client can assume the TURN server to support long-term auth, no need to check/negotiate that in any way.

Hmm, I changed my port yesterday but am still seeing many requests a second to 3478

That's because of what what someone else postet > because the attackers only scan once or every few months. They don't get an idea if your IP is still a usable reflector, only if they scan again.

Until you are out of every list they collected it will continue

So now you are doing your part in diluting the attack 😀

Wasting resources of the attackers

so with normal STUN usage is it request> response > request > response ? or just a single request > response ?

I don't see why they can't request continously...

Licaon_Kter, because if the protocol expects a second response, you can identify clients that don't do this and ignore them (preventing spoofing)

moparisthebest, with *un*authenticated *STUN*, it is request > response; with authenticated STUN it is request > response > request > response.

but AIUI authenticated STUN response on missing auth is (a) required for legitimate clients to work and (b) larger.

Just scored colloquy.ca 🤓 now I have a generic domain for friends ✏

slightly offtopic: loading a large ipset on fedora/rh causes each cidr insert to be checked against the selinux policy and not completely inserted. a list with a few thousand cidr's quickly ends up using 20+GB of ram. anyone have any insight about htis?

the same script works on debian, and only uses <200MB ram

at least that has an easy solution: turn off selinux