XMPP Service Operators - 2023-10-20

  1. ValdikSS

    Hello everyone. First of all, no links (archive or subscription) of the mailing lists are working https://xmpp.org/community/mailing-lists/

  2. ValdikSS


  3. ValdikSS

    And second, this might be interesting for every XMPP server operator: https://notes.valdikss.org.ru/jabber.ru-mitm/

  4. MattJ

    Thanks ValdikSS. Mailing list issue is known, and we're (gradually) working on it

  5. ValdikSS

    TL;DR: we have discovered XMPP (Jabber) instant messaging protocol encrypted TLS connection wiretapping (Man-in-the-Middle attack) of jabber.ru (aka xmpp.ru) service’s servers on Hetzner and Linode hosting providers in Germany. The attacker has issued several new TLS certificates using Let’s Encrypt service which were used to hijack encrypted STARTTLS connections on port 5222 using transparent MiTM proxy. The attack was discovered due to expiration of one of the MiTM certificates, which haven’t been reissued. There are no indications of the server breach or spoofing attacks on the network segment, quite the contrary: the traffic redirection has been configured on the hosting provider network. The wiretapping may have lasted for up to 6 months overall (90 days confirmed). We believe this is lawful interception Hetzner and Linode were forced to setup.

  6. Licaon_Kter

    (╯°□°)╯︵ ┻━┻

  7. Licaon_Kter

    They hijacked DNS too? How to get certs?

  8. MattJ

    ValdikSS, note that some XMPP clients support "channel binding", which would detect such MITM

  9. Guus

    You are saying that you think some kind of law enforcement agency has required hosting providers to set up MITM proxies for certain XMPP domains?

  10. ValdikSS

    > They hijacked DNS too? How to get certs? Over port 80 I guess

  11. MattJ

    Licaon_Kter, no need to hijack DNS, since the IP pointed to their middlebox

  12. ValdikSS

    > You are saying that you think some kind of law enforcement agency has required hosting providers to set up MITM proxies for certain XMPP domains? Yes, currently we assume that was the part of https://therecord.media/genesis-market-takedown-cybercrime

  13. Licaon_Kter

    There are several ways to generate certs, port 80 control is only part of it, MattJ

  14. Licaon_Kter

    Afaik, I'm a n00b anyway, happy to learn

  15. MattJ

    Licaon_Kter, I'm too familiar with the ways to obtain certs, but I don't think what you said contradicts what I said, so I'm not sure I understand the point you're making

  16. Licaon_Kter

    So, if you need to put entries in DNS to get LE to ack you, they need DNS access too. If they didn't use than path to generate, yes, etc

  17. ValdikSS

    We've sent the following email snippet to Hetzner If you can't comment on this matter any more, I will assume that the interception was set up by Hetzner due to legal requirements which can't be disclosed. Otherwise you should take action immediately and investigate your network. And they reply I refer to Ticket# 2023101703000688. Unfortunately, we cannot suggest any further solutions here. I understand from your reply that the problem has been solved.

  18. MSavoritias (fae,ve)

    well that sounds suspicious

  19. Licaon_Kter

    Now, if DNS is handled by Hetzner too, then all bets are off lol

    🙃 1
  20. savagepeanut

    You can require DNS validation with LE with a CAA record I think

  21. savagepeanut

    > Now, if DNS is handled by Hetzner too, then all bets are off lol 🙃

  22. MattJ

    Yeah, CAA record restricting to DNS challenges would have made an attack harder

  23. savagepeanut

    When will Snikket support DNS validation /j

  24. ValdikSS

    1. CAA restriction 2. Certificate transparency monitoring Unfortunately, none of these were configured when the attack started

  25. MattJ

    savagepeanut, both that and some kind of cert transparency monitoring has been on my to-do for a while. Partly because I want to make it easier for people to host their servers behind firewalls, but if DNS points at us we can easily obtain certs for their domain even if they terminate TLS. So I'd like to provide some assurance to people that's not happening.

  26. savagepeanut

    Ah, I wasn't thinking about the hosting service. That would make it even nicer to have then

  27. ValdikSS


  28. sagaracharya

    Anyone uses perdition?

  29. MattJ

    AFAICT certspotter doesn't differentiate between certs issued to you, or to someone else. So it would just notify you every month or so about your own certificates. That would be annoying, and make it more likely that you would miss actual issues.

  30. RTG

    > Yeah, CAA record restricting to DNS challenges would have made an attack harder Keep in mind some registrars that host the domain's DNS can provide CAA records that are not visible in the customer's control panel. So to make the attack even harder (albeit not impossible), host one's own DNS with DNSSEC. :)

  31. ValdikSS

    You could also limit the validation methods and even account ID for ACME-based services https://datatracker.ietf.org/doc/rfc8657/

  32. array

    xmpp is **FED**erated

  33. array

    xmpp is **FED** erated

  34. ValdikSS

    MattJ, care to elaborate more in simplier terms regarding SCRAM for me to add this to the blog post?

  35. ValdikSS

    Sorry, don't have time to read the RFC myself. I assume this is a kind of key exchange method which involve public server certificate from the server side, and if it doesn't match for both client and server, the validation fails?

  36. MattJ

    Pretty much, yes

  37. MattJ

    It doesn't have to use the certificate, but that is one option

  38. MattJ

    I did post https://news.ycombinator.com/item?id=37957677 with a bit more explanation

  39. ValdikSS

    Could you please write it in one-two sentences in a form of which is already present in the very bottom of the post please?

  40. ValdikSS

    "Could you prevent or monitor this kind of attack?" section

  41. MattJ

    Well, jabber.ru doesn't seem to offer it currently, so there is nothing users can do

  42. MattJ

    I'm not sure why it is not offered, I'm fairly sure ejabberd has supported it for years

  43. jonas’

    ValdikSS, actually you should probably recommend users to change their passwords immediately, if you hadn't offered SCRAM before.

  44. ValdikSS

    Let me just invite the admin here

  45. oxpa

    Hello everyone

  46. ValdikSS

    > ValdikSS, actually you should probably recommend users to change their passwords immediately, if you hadn't offered SCRAM before.

  47. MattJ

    But for your request, perhaps the following text: "'Channel binding' is a feature in XMPP which can detect a MITM even if they present a valid certificate. Both the client and the server must support SCRAM PLUS authentication mechanisms for this to work. Unfortunately this was not active on jabber.ru at the time of the attack."

  48. ValdikSS

    MattJ, thanks!

  49. ValdikSS

    jonas’, changing password is more complicated in jabber.ru than on most other servers, oxpa could elaborate why. It require email, but many jabber.ru users don't have email filled.

  50. oxpa

    It's just due to historical reasons. Many users /can/ change their password through a client

  51. ValdikSS

    oxpa, and MattJ told me that Channel binding feature could have prevented this kind of attack for the clients which support it, but it's not enabled on jabber.ru

  52. oxpa

    jabber.ru runs a very old version of ejabberd. We are working to upgrade it (but were too late, it seems)

  53. oxpa

    it's the first time I hear about channel binding. I'll read about it and make sure to enable it when possible

  54. MattJ

    I know how that feels (jabber.org...) 😅

  55. oxpa

    for now, I have CAA added and ermine (she owns DNS server) works on setting up DNS SEC

  56. oxpa

    that should make things a bit harder for any attacker

  57. oxpa

    I just doubt that all CAs support CAA and DNS SEC (but I have hopes =)

  58. savagepeanut

    https://desec.io/ looks pretty good if you're interested in hosting your own DNS. I've been meaning to move to it because porkbun uses cloudflare for dnssec and they add a bunch of CAA records

  59. jonas’

    (note that desec.io is based in germany which, while a Good Thing™ from certain perspectives, in this particular instance migth not be the best choice)

  60. savagepeanut


  61. oxpa

    I have my own DNS servers and I also have a plan to harden them a bit (i guess it's called "shadow master"). Cause at the moment one of the server /is/ the real master

  62. oxpa

    I guess, I want to avoid this

  63. oxpa

    I'd like to note that "channel binding" is more specialised solution. CAA + DNSSEC + better monitoring is more general purpose

  64. MattJ

    CAA + DNSSEC + DANE with the client validating DNSSEC would indeed be secure and great

  65. MattJ

    Channel binding is easier to deploy and usable today

  66. oxpa

    and you don't really need /all/ clients to do this. CA would be enough to avoid the main issue: malicious certs

  67. MattJ

    Unless the CA is requested to ignore the CAA

  68. jonas’

    MattJ, easier to deploy if your ejabberd is too ancient for that?

  69. MattJ

    jonas’: no, but that's one thing to upgrade vs every DNS resolver your clients might end up using

  70. jonas’

    yep sure

  71. oxpa

    MattJ: let's say, i'd be happy to prove a CA ignored CAA when it tells CAA are honored. That would put that CA out of business pretty quickly. Though this won't work with LE: too big, too popular

  72. MattJ

    You could argue that a hosting provider silently MITMing their customers' encrypted traffic should also put them out of business quickly 🙂

  73. savagepeanut

    They should sell it as a service like cloudflare ;)

  74. jonas’


  75. MattJ

    But both cases would be organizations legally required to do what they were ordered

  76. oxpa

    MattJ: can't argue with that =)

  77. jonas’

    though it would likely be harder to do so across the jurisdictions involved in the case of hetzner + Let's encrypt

  78. oxpa

    It would be really nice to force Hetzner disclose details of these actions. Is there an order? Or is it just 'scan everything' ?

  79. ValdikSS

    oxpa, there should be, see https://www.bfdi.bund.de/EN/Buerger/Inhalte/Nachrichtendienste/Telekommunikationsueberwachung.html

  80. jonas’

    oxpa, other dedicated and virtual servers on hetzner are not affected (I was not able to reproduce the changed port numbers for instance), so I suspect that this was somewhat targeted. and hetzner's pricing being what it is, I doubt they do this without being forced to.

  81. ValdikSS

    You probably should contact the lawyer and send them a paper letter, that would increase the chance for a proper rely.

  82. ValdikSS

    You probably should contact the lawyer and send them a paper letter, that would increase the chance for a proper reply.

  83. ValdikSS

    Both to hetzner and linode

  84. oxpa

    well, I have to find a lawyer to start with. Maybe I can get a discount, given that their cert expired =)

  85. MattJ

    Good luck! Will be curious to hear how it goes, if you really try

  86. ValdikSS

    Basically we sent the support tickets with the text that implies if we can't receive any further communication on this issue from them we will assume that they can't comment due to legal obligations.

  87. ValdikSS

    And we haven't received anything from Linode yet and only a simple reply from Hetzner

  88. oxpa

    simple reply that reads "we can't comment further"

  89. oxpa

    also, would be fun to sue german police for an expired cert

  90. ru_maniac

    oxpa, any chance that you would consider using commercial-grade certificate from now on?

  91. ru_maniac

    i.e., Let's Encrypt hijack is one thing

  92. ru_maniac

    but attack surface should be much smaller

  93. ru_maniac

    if a big CA would spoof the certs, that would be a shitstorm in itself

  94. Link Mauve

    ru_maniac, no client ever checks for EV, do they?

  95. oxpa

    ru_maniac: let;s say I got a commercial one. MitM still has LE. Nothing changes, does it?

  96. Martin

    Anyone knows the jabberes admin? There is no 0157 and also no contact info on https://www.jabberes.org/en/.

  97. jonas’

    oxpa, if you set your CAA to the commercial one, LE should not issue certificates.

  98. jonas’

    makes the attack much harder

  99. ru_maniac

    oxpa, you would've spotted that immediately

  100. ru_maniac

    if your CAA would ve been configured correctly

  101. oxpa

    well, with proper caa and monitoring LE is also good enough

  102. ru_maniac

    Link Mauve, that's not a client issue tbh

  103. oxpa

    so still not quite clear what's the reason to have another CA

  104. ru_maniac

    LE allows to issue certs automatically

  105. ru_maniac

    should they intercept 80 port or acquire a token off DNS API which allows to issue certs via DNS challenge -- they can issue however many certs they want

  106. ru_maniac

    LE allows multiple certs to exist for a single TLD, no problem

  107. ru_maniac

    but, if a commercial CA were to issue a certificate, third party would have to deal with them -- which would end their reputation in case they comply

  108. ru_maniac

    oxpa, btw, if you don't mind -- I have one last question would you consider to finally implement 0384 on your server?

  109. ru_maniac

    it's been years

  110. ru_maniac

    I can live with the idea of plain c2s or s2s communications, this is third-party servers issue tbh but no OMEMO support is a ouchie

  111. ru_maniac

    I can live with the idea of plain c2s or s2s communications, this is third-party servers issue tbh but no OMEMO support is an ouchie

  112. oxpa

    ru_maniac: you can move a host that aquires certs to the dns server itself . And then automate publishing certs to worker nodes (ejabberd, nginx, whatever). It can be done pretty secure.

  113. oxpa

    commercial certificate only works "better" if one have caa, dnssec and all other things in place

  114. oxpa

    in that scenario LE is already good enough

  115. oxpa

    and I don't remember what is 0384 but most likely it doesn't work on ejabberd from 2016 so that's why it is not yet implemented on jru

  116. oxpa

    we slowly work (i'd say firmly lay) toward ejabberd upgrade to a semi modern version to migrate to a modern version after that

  117. oxpa

    it's not like I have anything against OMEMO. It just doesn't work on the current ejabberd)

  118. RTG

    > It's just due to historical reasons. Many users /can/ change their password through a client If this was indeed lawful interception of which the hosting providers complied with, authorities may have extracted a disk image in addition to the MiTM. Without SCRAM, I'd increase the necessity of changing passwords. :)

  119. ru_maniac

    oxpa, 0384 is a OMEMO encryption XEP the reason I'm asking is that I vaguely remember that twitter account associated with jru wrote something a while back re: how encryption is not really required and that STARTTLS for s2s and c2s won't be implemented, period

  120. oxpa

    RTG: we notified all users that there was mitm, passwords must be changed and messages considered compromised

  121. oxpa

    ru_maniac: can you send a link? I own that account and I shouldn't have posted anything like that

  122. ru_maniac

    I'm scrolling thru it right now

  123. oxpa

    ru_maniac: imo, "my server is secure" when you are talking about a public service provider, is a false sense of security

  124. oxpa

    a user has to estimate the risk

  125. ru_maniac

    estimation is one thing, mocking the idea of transit encryption is another

  126. oxpa

    it doesn't mean though that starttls is not needed

  127. sagaracharya

    Anyone uses popa3d?

  128. ru_maniac

    https://twitter.com/JabberRu/status/1128578741016330240 https://twitter.com/JabberRu/status/1032012649561763840 https://twitter.com/JabberRu/status/1014181486256418817 https://twitter.com/JabberRu/status/905345807544659968

  129. ru_maniac

    those are the posts that I've seen (and archived on web.archive.org, so no point in deleting them), outright mocking the "crypto nerds"

  130. oxpa

    ru_maniac: in the third tweet I mess up ISS and ISIS, to be honest=)

  131. ru_maniac

    maybe I'm reading way between the lines, but it seems to me that you're trying to communicate "if you don't have anything to hide, there's nothing to be afraid of" point of view

  132. oxpa

    but I won't stop making fun of cryptonerds. Quite a few of them knows no basics at all and can only mock other for not using "this new awesome tool"

  133. ru_maniac

    one might make fun of whomever they want, but I believe that best practices are, well, quite established

  134. ru_maniac

    so maybe it's time for transit encryption for c2s and s2s, it's been ten years

    🚂 1
  135. oxpa

    s/transit/transport/ ?

  136. ru_maniac

    yup, typo

  137. savagepeanut

    > so maybe it's time for transit encryption for c2s and s2s, it's been ten years 🚂

  138. oxpa

    ru_maniac: jabber.ru 5269 supports starttls. Do you want anything more?

  139. ru_maniac

    it supports, but it doesn't force other servers to communicate via TLS only

  140. ru_maniac

    it supports it, but it doesn't force other servers to communicate via TLS only

  141. ru_maniac

    I mean, I can't think of a public XMPP service today which is not using TLS for s2s

  142. ru_maniac

    the same goes for c2s, tbh

  143. ru_maniac

    I mean, personally I don't really care, I run my own server -- it's just a point of principle to me, since some of my acquaintances are still using your service

  144. oxpa

    ru_maniac: enforce tls on your own server. That will solve the problem for you. Last time i disabled cleartext on c2s - a ton of bot users spammed my jid. I don't think it got any better since)

  145. oxpa

    more or less the same goes for c2s. I'll give it a thought though. shouldn't be a huge problem

  146. nuegia.net

    oxpa, what happened?

  147. ru_maniac

    oxpa, no need to get defensive here -- this is an ask, not a demand. your service, your rules, your responsibility. thanks for at least listening to me here.

  148. ru_maniac

    nuegia.net, seems like Hetzner and Linode have been forced to implement MiTM attack against jabber.ru and xmpp.ru service

  149. ru_maniac

    full write-up is here: https://notes.valdikss.org.ru/jabber.ru-mitm/

  150. nuegia.net

    thank you

  151. nuegia.net

    I want to say that I have been having trouble with Linode hosting people who have been constantly harassing and attacking my users and I, including spamming links to child pornography on my servers. I have been repeatily reporting these issues to Linode however their "trust & safety team" ignores evidence and logs presented to them and an unwillingness to respond to any kind of abuse besides removing a web page served over http.

  152. nuegia.net

    Linode has flat out denied that they owned IP addresses involved in these attacks despite WHOIS data and the ASN showing otherwise

  153. octagon

    reminder to setup CAA https://sslmate.com/caa/ and DNSSEC

  154. octagon

    and maybe onions should become more popular: https://matt.traudt.xyz/posts/2022-11-09-tor-is-not-just-for-anonymity/

  155. ru_maniac

    onion-enabled hosts are a fun notion, but all in all, even less user-friendly than regular xmpp per se

  156. ru_maniac

    (in my opinion)

  157. octagon

    CAA records can and should also be bound to a single LE accounturi, it isn't really documented so its like issue "letsencrypt.org; validationmethods=http-01; accounturi=https://acme-v02.api.letsencrypt.org/acme/acct/ACCOUNTNUMBER" replace the number and api endpoint from whereever it is saved in /etc/letsencrypt or whatever

  158. octagon

    you can also setup the iodef rule to email you

  159. ru_maniac

    if there are multiple shards\VMs, it can be a bit tricky to tie all of certbot/acme.sh entities to a single account

  160. octagon

    you can add multiple

  161. octagon

    just repeat the `issue` line

  162. octagon

    it handles just fine

  163. octagon

    still better to allow 5 accounts than all

  164. ru_maniac

    oh, that's a nice trick then, thanks for bringing that up

  165. ru_maniac

    that is true

  166. octagon

    also, this brings up a point of clients actually validating dnssec conversations does not by default and most resovlers don't either

  167. octagon

    oxpa, if you have evidence of a CA ignoring CAA, then you can report them to the CAB

  168. savagepeanut

    It is tricky to by default with TLDs in use that don't support dnssec at all, like .im

  169. oxpa

    octagon: no-no, no evidence. It would be fun to have any =) But I have no doubts in LE at the moment, nothing to report

  170. oxpa

    octagon: you really need dns sec for CA's resolvers. If I'd setup CAA and DNS SEC earlier the whole thing would not be possible

  171. oxpa

    well, possible but not as easily

  172. ru_maniac

    savegepeanut, fortunately .ru TLD supports DNSSEC, so for this service in particular this is a non-issue

  173. nuegia.net

    which region were the compromised servers in?

  174. oxpa

    frankfurt (both linode and hetzner)

  175. savagepeanut

    > savegepeanut, fortunately .ru TLD supports DNSSEC, so for this service in particular this is a non-issue That is fortunate, but I meant having clients verify it by default in general

  176. oxpa

    it will be funny if linode servers are hosted in hetzner, btw. IDK where they are physically

  177. ru_maniac

    savagepeanut, I don't know many clients whom verify DNSSEC records by default

  178. ru_maniac

    maybe Conversations? Dino does not seem to support that

  179. nuegia.net

    I really appreciate this writeup and hardening advice

  180. savagepeanut

    Conversations does not

  181. octagon

    conversations has it under settings > expert settings > validate hostname with dnssec maybe daniel can be pushed to enabling it by default

  182. ru_maniac

    this is a bad idea, not every DNS resolver supports that

  183. savagepeanut

    Right :(

  184. ru_maniac

    if your ISP box is not configurable, and relies solely on ISPs DNS service, this may break things

  185. Menel

    And not every domain

  186. octagon

    ru_maniac, it has a safe fallback if the resolver doesn't suport it

  187. octagon

    and it doens't hard fail if the domain doesn't use dnssec

  188. Menel

    .im doesn't AFAIK

  189. ru_maniac

    octagon, then it kinda negates the point, doesn't it?

  190. octagon

    it can still slightly raise the bar

  191. nuegia.net

    this attack the way it was performed reminds me a lot of the way the lavabit and kim.com attacks were made

  192. octagon

    that is all security is, making it cost more time or $$$

  193. nuegia.net

    does prosody support channel binding?

  194. savagepeanut