XMPP Service Operators - 2023-10-20

Hello everyone. First of all, no links (archive or subscription) of the mailing lists are working https://xmpp.org/community/mailing-lists/

1

And second, this might be interesting for every XMPP server operator: https://notes.valdikss.org.ru/jabber.ru-mitm/

Thanks ValdikSS. Mailing list issue is known, and we're (gradually) working on it

TL;DR: we have discovered XMPP (Jabber) instant messaging protocol encrypted TLS connection wiretapping (Man-in-the-Middle attack) of jabber.ru (aka xmpp.ru) service’s servers on Hetzner and Linode hosting providers in Germany. The attacker has issued several new TLS certificates using Let’s Encrypt service which were used to hijack encrypted STARTTLS connections on port 5222 using transparent MiTM proxy. The attack was discovered due to expiration of one of the MiTM certificates, which haven’t been reissued. There are no indications of the server breach or spoofing attacks on the network segment, quite the contrary: the traffic redirection has been configured on the hosting provider network. The wiretapping may have lasted for up to 6 months overall (90 days confirmed). We believe this is lawful interception Hetzner and Linode were forced to setup.

(╯°□°）╯︵ ┻━┻

They hijacked DNS too? How to get certs?

ValdikSS, note that some XMPP clients support "channel binding", which would detect such MITM

You are saying that you think some kind of law enforcement agency has required hosting providers to set up MITM proxies for certain XMPP domains?

> They hijacked DNS too? How to get certs? Over port 80 I guess

Licaon_Kter, no need to hijack DNS, since the IP pointed to their middlebox

> You are saying that you think some kind of law enforcement agency has required hosting providers to set up MITM proxies for certain XMPP domains? Yes, currently we assume that was the part of https://therecord.media/genesis-market-takedown-cybercrime

There are several ways to generate certs, port 80 control is only part of it, MattJ

Afaik, I'm a n00b anyway, happy to learn

Licaon_Kter, I'm too familiar with the ways to obtain certs, but I don't think what you said contradicts what I said, so I'm not sure I understand the point you're making

So, if you need to put entries in DNS to get LE to ack you, they need DNS access too. If they didn't use than path to generate, yes, etc

We've sent the following email snippet to Hetzner If you can't comment on this matter any more, I will assume that the interception was set up by Hetzner due to legal requirements which can't be disclosed. Otherwise you should take action immediately and investigate your network. And they reply I refer to Ticket# 2023101703000688. Unfortunately, we cannot suggest any further solutions here. I understand from your reply that the problem has been solved.

well that sounds suspicious

Now, if DNS is handled by Hetzner too, then all bets are off lol

You can require DNS validation with LE with a CAA record I think

> Now, if DNS is handled by Hetzner too, then all bets are off lol 🙃 ↺

Yeah, CAA record restricting to DNS challenges would have made an attack harder

When will Snikket support DNS validation /j

1. CAA restriction 2. Certificate transparency monitoring Unfortunately, none of these were configured when the attack started

savagepeanut, both that and some kind of cert transparency monitoring has been on my to-do for a while. Partly because I want to make it easier for people to host their servers behind firewalls, but if DNS points at us we can easily obtain certs for their domain even if they terminate TLS. So I'd like to provide some assurance to people that's not happening.

Ah, I wasn't thinking about the hosting service. That would make it even nicer to have then

https://github.com/SSLMate/certspotter

Anyone uses perdition?

AFAICT certspotter doesn't differentiate between certs issued to you, or to someone else. So it would just notify you every month or so about your own certificates. That would be annoying, and make it more likely that you would miss actual issues.

> Yeah, CAA record restricting to DNS challenges would have made an attack harder Keep in mind some registrars that host the domain's DNS can provide CAA records that are not visible in the customer's control panel. So to make the attack even harder (albeit not impossible), host one's own DNS with DNSSEC. :) ↺

You could also limit the validation methods and even account ID for ACME-based services https://datatracker.ietf.org/doc/rfc8657/

xmpp is **FED** erated ✏

MattJ, care to elaborate more in simplier terms regarding SCRAM for me to add this to the blog post?

Sorry, don't have time to read the RFC myself. I assume this is a kind of key exchange method which involve public server certificate from the server side, and if it doesn't match for both client and server, the validation fails?

Pretty much, yes

It doesn't have to use the certificate, but that is one option

I did post https://news.ycombinator.com/item?id=37957677 with a bit more explanation

Could you please write it in one-two sentences in a form of which is already present in the very bottom of the post please?

"Could you prevent or monitor this kind of attack?" section

Well, jabber.ru doesn't seem to offer it currently, so there is nothing users can do

I'm not sure why it is not offered, I'm fairly sure ejabberd has supported it for years

ValdikSS, actually you should probably recommend users to change their passwords immediately, if you hadn't offered SCRAM before.

Let me just invite the admin here

Hello everyone

> ValdikSS, actually you should probably recommend users to change their passwords immediately, if you hadn't offered SCRAM before.

But for your request, perhaps the following text: "'Channel binding' is a feature in XMPP which can detect a MITM even if they present a valid certificate. Both the client and the server must support SCRAM PLUS authentication mechanisms for this to work. Unfortunately this was not active on jabber.ru at the time of the attack."

MattJ, thanks!

jonas’, changing password is more complicated in jabber.ru than on most other servers, oxpa could elaborate why. It require email, but many jabber.ru users don't have email filled.

It's just due to historical reasons. Many users /can/ change their password through a client

oxpa, and MattJ told me that Channel binding feature could have prevented this kind of attack for the clients which support it, but it's not enabled on jabber.ru

jabber.ru runs a very old version of ejabberd. We are working to upgrade it (but were too late, it seems)

it's the first time I hear about channel binding. I'll read about it and make sure to enable it when possible

I know how that feels (jabber.org...) 😅

for now, I have CAA added and ermine (she owns DNS server) works on setting up DNS SEC

that should make things a bit harder for any attacker

I just doubt that all CAs support CAA and DNS SEC (but I have hopes =)

https://desec.io/ looks pretty good if you're interested in hosting your own DNS. I've been meaning to move to it because porkbun uses cloudflare for dnssec and they add a bunch of CAA records

(note that desec.io is based in germany which, while a Good Thing™ from certain perspectives, in this particular instance migth not be the best choice)

🫠

I have my own DNS servers and I also have a plan to harden them a bit (i guess it's called "shadow master"). Cause at the moment one of the server /is/ the real master

I guess, I want to avoid this

I'd like to note that "channel binding" is more specialised solution. CAA + DNSSEC + better monitoring is more general purpose

CAA + DNSSEC + DANE with the client validating DNSSEC would indeed be secure and great

Channel binding is easier to deploy and usable today

and you don't really need /all/ clients to do this. CA would be enough to avoid the main issue: malicious certs

Unless the CA is requested to ignore the CAA

MattJ, easier to deploy if your ejabberd is too ancient for that?

jonas’: no, but that's one thing to upgrade vs every DNS resolver your clients might end up using

yep sure

MattJ: let's say, i'd be happy to prove a CA ignored CAA when it tells CAA are honored. That would put that CA out of business pretty quickly. Though this won't work with LE: too big, too popular

You could argue that a hosting provider silently MITMing their customers' encrypted traffic should also put them out of business quickly 🙂

They should sell it as a service like cloudflare ;)

hah

But both cases would be organizations legally required to do what they were ordered

MattJ: can't argue with that =)

though it would likely be harder to do so across the jurisdictions involved in the case of hetzner + Let's encrypt

It would be really nice to force Hetzner disclose details of these actions. Is there an order? Or is it just 'scan everything' ?

oxpa, there should be, see https://www.bfdi.bund.de/EN/Buerger/Inhalte/Nachrichtendienste/Telekommunikationsueberwachung.html

oxpa, other dedicated and virtual servers on hetzner are not affected (I was not able to reproduce the changed port numbers for instance), so I suspect that this was somewhat targeted. and hetzner's pricing being what it is, I doubt they do this without being forced to.

You probably should contact the lawyer and send them a paper letter, that would increase the chance for a proper reply. ✏

Both to hetzner and linode

well, I have to find a lawyer to start with. Maybe I can get a discount, given that their cert expired =)

Good luck! Will be curious to hear how it goes, if you really try

Basically we sent the support tickets with the text that implies if we can't receive any further communication on this issue from them we will assume that they can't comment due to legal obligations.

And we haven't received anything from Linode yet and only a simple reply from Hetzner

simple reply that reads "we can't comment further"

also, would be fun to sue german police for an expired cert

oxpa, any chance that you would consider using commercial-grade certificate from now on?

i.e., Let's Encrypt hijack is one thing

but attack surface should be much smaller

if a big CA would spoof the certs, that would be a shitstorm in itself

ru_maniac, no client ever checks for EV, do they?

ru_maniac: let;s say I got a commercial one. MitM still has LE. Nothing changes, does it?

Anyone knows the jabberes admin? There is no 0157 and also no contact info on https://www.jabberes.org/en/.

oxpa, if you set your CAA to the commercial one, LE should not issue certificates.

makes the attack much harder

oxpa, you would've spotted that immediately

if your CAA would ve been configured correctly

well, with proper caa and monitoring LE is also good enough

Link Mauve, that's not a client issue tbh

so still not quite clear what's the reason to have another CA

LE allows to issue certs automatically

should they intercept 80 port or acquire a token off DNS API which allows to issue certs via DNS challenge -- they can issue however many certs they want

LE allows multiple certs to exist for a single TLD, no problem

but, if a commercial CA were to issue a certificate, third party would have to deal with them -- which would end their reputation in case they comply

oxpa, btw, if you don't mind -- I have one last question would you consider to finally implement 0384 on your server?

it's been years

I can live with the idea of plain c2s or s2s communications, this is third-party servers issue tbh but no OMEMO support is an ouchie ✏

ru_maniac: you can move a host that aquires certs to the dns server itself . And then automate publishing certs to worker nodes (ejabberd, nginx, whatever). It can be done pretty secure.

commercial certificate only works "better" if one have caa, dnssec and all other things in place

in that scenario LE is already good enough

and I don't remember what is 0384 but most likely it doesn't work on ejabberd from 2016 so that's why it is not yet implemented on jru

we slowly work (i'd say firmly lay) toward ejabberd upgrade to a semi modern version to migrate to a modern version after that

it's not like I have anything against OMEMO. It just doesn't work on the current ejabberd)

> It's just due to historical reasons. Many users /can/ change their password through a client If this was indeed lawful interception of which the hosting providers complied with, authorities may have extracted a disk image in addition to the MiTM. Without SCRAM, I'd increase the necessity of changing passwords. :) ↺

oxpa, 0384 is a OMEMO encryption XEP the reason I'm asking is that I vaguely remember that twitter account associated with jru wrote something a while back re: how encryption is not really required and that STARTTLS for s2s and c2s won't be implemented, period

RTG: we notified all users that there was mitm, passwords must be changed and messages considered compromised

ru_maniac: can you send a link? I own that account and I shouldn't have posted anything like that

I'm scrolling thru it right now

ru_maniac: imo, "my server is secure" when you are talking about a public service provider, is a false sense of security

a user has to estimate the risk

estimation is one thing, mocking the idea of transit encryption is another

it doesn't mean though that starttls is not needed

Anyone uses popa3d?

https://twitter.com/JabberRu/status/1128578741016330240 https://twitter.com/JabberRu/status/1032012649561763840 https://twitter.com/JabberRu/status/1014181486256418817 https://twitter.com/JabberRu/status/905345807544659968

those are the posts that I've seen (and archived on web.archive.org, so no point in deleting them), outright mocking the "crypto nerds"

ru_maniac: in the third tweet I mess up ISS and ISIS, to be honest=)

maybe I'm reading way between the lines, but it seems to me that you're trying to communicate "if you don't have anything to hide, there's nothing to be afraid of" point of view

but I won't stop making fun of cryptonerds. Quite a few of them knows no basics at all and can only mock other for not using "this new awesome tool"

one might make fun of whomever they want, but I believe that best practices are, well, quite established

so maybe it's time for transit encryption for c2s and s2s, it's been ten years

s/transit/transport/ ?

yup, typo

> so maybe it's time for transit encryption for c2s and s2s, it's been ten years 🚂 ↺

ru_maniac: jabber.ru 5269 supports starttls. Do you want anything more?

it supports it, but it doesn't force other servers to communicate via TLS only ✏

I mean, I can't think of a public XMPP service today which is not using TLS for s2s

the same goes for c2s, tbh

I mean, personally I don't really care, I run my own server -- it's just a point of principle to me, since some of my acquaintances are still using your service

ru_maniac: enforce tls on your own server. That will solve the problem for you. Last time i disabled cleartext on c2s - a ton of bot users spammed my jid. I don't think it got any better since)

more or less the same goes for c2s. I'll give it a thought though. shouldn't be a huge problem

oxpa, what happened?

oxpa, no need to get defensive here -- this is an ask, not a demand. your service, your rules, your responsibility. thanks for at least listening to me here.

nuegia.net, seems like Hetzner and Linode have been forced to implement MiTM attack against jabber.ru and xmpp.ru service

full write-up is here: https://notes.valdikss.org.ru/jabber.ru-mitm/

thank you

I want to say that I have been having trouble with Linode hosting people who have been constantly harassing and attacking my users and I, including spamming links to child pornography on my servers. I have been repeatily reporting these issues to Linode however their "trust & safety team" ignores evidence and logs presented to them and an unwillingness to respond to any kind of abuse besides removing a web page served over http.

Linode has flat out denied that they owned IP addresses involved in these attacks despite WHOIS data and the ASN showing otherwise

reminder to setup CAA https://sslmate.com/caa/ and DNSSEC

and maybe onions should become more popular: https://matt.traudt.xyz/posts/2022-11-09-tor-is-not-just-for-anonymity/

onion-enabled hosts are a fun notion, but all in all, even less user-friendly than regular xmpp per se

(in my opinion)

CAA records can and should also be bound to a single LE accounturi, it isn't really documented so its like issue "letsencrypt.org; validationmethods=http-01; accounturi=https://acme-v02.api.letsencrypt.org/acme/acct/ACCOUNTNUMBER" replace the number and api endpoint from whereever it is saved in /etc/letsencrypt or whatever

you can also setup the iodef rule to email you

if there are multiple shards\VMs, it can be a bit tricky to tie all of certbot/acme.sh entities to a single account

you can add multiple

just repeat the `issue` line

it handles just fine

still better to allow 5 accounts than all

oh, that's a nice trick then, thanks for bringing that up

that is true

also, this brings up a point of clients actually validating dnssec conversations does not by default and most resovlers don't either

oxpa, if you have evidence of a CA ignoring CAA, then you can report them to the CAB

It is tricky to by default with TLDs in use that don't support dnssec at all, like .im

octagon: no-no, no evidence. It would be fun to have any =) But I have no doubts in LE at the moment, nothing to report

octagon: you really need dns sec for CA's resolvers. If I'd setup CAA and DNS SEC earlier the whole thing would not be possible

well, possible but not as easily

savegepeanut, fortunately .ru TLD supports DNSSEC, so for this service in particular this is a non-issue

which region were the compromised servers in?

frankfurt (both linode and hetzner)

> savegepeanut, fortunately .ru TLD supports DNSSEC, so for this service in particular this is a non-issue That is fortunate, but I meant having clients verify it by default in general ↺

it will be funny if linode servers are hosted in hetzner, btw. IDK where they are physically

savagepeanut, I don't know many clients whom verify DNSSEC records by default

maybe Conversations? Dino does not seem to support that

I really appreciate this writeup and hardening advice

Conversations does not

conversations has it under settings > expert settings > validate hostname with dnssec maybe daniel can be pushed to enabling it by default

this is a bad idea, not every DNS resolver supports that

Right :(

if your ISP box is not configurable, and relies solely on ISPs DNS service, this may break things

And not every domain

ru_maniac, it has a safe fallback if the resolver doesn't suport it

and it doens't hard fail if the domain doesn't use dnssec

.im doesn't AFAIK

octagon, then it kinda negates the point, doesn't it?

it can still slightly raise the bar

this attack the way it was performed reminds me a lot of the way the lavabit and kim.com attacks were made

that is all security is, making it cost more time or $$$

does prosody support channel binding?

Yes