XMPP Service Operators - 2021-04-29

  1. Martin

    WTF Melinda Thompson?

  2. jonas’

    I, too, have questions.

  3. Maranda

    Poor jonas’... Being sued for talking about STUN/TURN security implications on a mailing list

  4. Maranda


  5. Licaon_Kter

    Maranda: links...links c'mon

  6. jonas’

    Licaon_Kter, https://mail.jabber.org/pipermail/operators/2021-April/003135.html

  7. Licaon_Kter


  8. Licaon_Kter

    Nothing says real person more than `"Jane Doe" <evesmith3413@gmail.com>` ¯\_(ツ)_/¯

  9. pepta.net

    Relaying e-mail to the police? Sounds like a compromised STUN over SMTP

  10. pepta.net

    Relaying e-mail to the police? Sounds like a compromised TURN over SMTP

  11. menel

    Anonymus threats is not the same as "beeing sued" That mail is laughable.

  12. jonas’

    very true. I’m not taking this in any way serious, esp. because there’s nothing I’ve done wrong under any jurisdiction affecting me.

  13. Martin just wonders what's wrong with that person.

  14. jonas’

    could be a translation issue or maybe they didn’t realize that the mail came from a mailing list or maybe google flagged it because DKIM and that made it look suspicious or whatever

  15. pepta.net

    Or this is very serious and sooner or later you'll receive an injonction from tribunal not to ask for people to randomize their STUN/TURN ports xD (joke)

  16. Licaon_Kter

    What a difference an year makes 2020: OMG https://nitter.fdn.fr/HCornflower/status/1254003453011034112#m & https://nitter.fdn.fr/HCornflower/status/1254068862158614529#m VS 2021: MEH https://mail.jabber.org/pipermail/operators/2021-April/003131.html

  17. Wiktor

    lol didn't know the guy so closely follows xmpp 😀

  18. octagon

    Is simply changing port number really that effective? Even if an enterprising attacker can port scan or lookup SRV records?

  19. Martin

    I don't have srv records for stun/turn

  20. jonas’

    Wiktor, you are aware that Philipp Hancke is fippo, who has a huge part in the authorship of Jingle?

  21. jonas’

    octagon, scanning all UDP ports + all IPv4 addresses is much more expensive (by a factor of 65535) than just scanning all IPv4 addresses on a well-known port. and yes, the STUN/TURN port is not advertised via SRV typically

  22. Wiktor

    jonas’, yeah, but I didn't know he's still interested in that in context of xmpp, I thought it's part of the disconnect of google from xmpp and just using jingle for other purposes (webrtc)

  23. jonas’

    he’s sometimes to be seen on the xeps github as well as standards@

  24. Wiktor

    oh, I see

  25. jonas’

    and, apparently, operators@

  26. Wiktor

    always criticizing... I get it 🙂

  27. jonas’

    and Licaon_Kter, I don’t blame him for questioning this... I am also fairly certain that the email has to be read as "are you sure this is being attacked? if so... we need to ask critically why, because it does no seem to be the amplification factor alone which makes this interesting".

  28. jonas’

    Wiktor, I’d call it "asking the hard questions nobody wants to think about" especially in the linked context

  29. Licaon_Kter

    Ofcourse, it just pinged when I saw his name and remembered those remarks on Conversations A/V launch.

  30. Licaon_Kter

    Yes, I want to see the links to the attacks too...as I don't see any random stun/turn connections here. But whenwI hosted SIP I had >1000 connections on 5060/5061, no visible impact but they were there in my router

  31. Licaon_Kter

    Yes, I want to see the links to the attacks too...as I don't see any random stun/turn connections here. But when I hosted SIP I had >1000 connections on 5060/5061, no visible impact but they were there in my router

  32. jonas’

    Licaon_Kter, how are you measuring your STUN/TURN "connections"?

  33. Licaon_Kter

    I see them in ejabberd logs, but was expecting activity on the router/firewall

  34. Licaon_Kter

    I'm not measuring, actively, but RPi's are rather sensible to load :)

  35. jonas’

    Licaon_Kter, maybe do a tcpdump on udp port 3478

  36. pepta.net

    What a mess nowadays to allow two peers to exchange data... STUN/TURNS/other protocols, server ressources... all because of ipv4 is still alive. Internet failure.

  37. Licaon_Kter

    All because ipv6 was just born we should do what? pepta.net

  38. pepta.net

    Us: nothing. But ICANN could have speed up things mandating every ISP to provide IPv6 everywhere and deprecate ipv4 with deadlines.

  39. pepta.net

    Us: nothing. But ICANN could have speed up things mandating every ISP to provide IPv6 everywhere and deprecate (and kill) ipv4 with deadlines.

  40. pepta.net

    Dual stack seems more now forever than a temporary transitional situation. Making everyone to suffer from ipv4 cons.

  41. Ge0rG


  42. pepta.net

    Except we can provide IPv6 compatible services whenever possible.

  43. pepta.net

    Except we can provide IPv6 compatible services whenever possible. To make things evolve step-by-step.

  44. Licaon_Kter

    Who's this we? How can I provide anything where there's no 6 here?

  45. pepta.net

    We = service providers. If no 6 available, can't ask for the impossible. Maybe consider ISP/hosting switching, or any action to manifest your disappointment to your providers.

  46. Licaon_Kter

    No 6 in sight... But see, they announced 6 back in '15 so PR side all is well

  47. moparisthebest

    Couldn't you filter the IPs you accept connections to your stun server on based on the IPs of clients currently connected to your XMPP server?

  48. jonas’

    moparisthebest, no, you also need to accept packets from their peers

  49. jonas’


  50. jonas’

    at least once the channel is established

  51. jonas’

    though possibly that is only after the initial STUN phase

  52. jonas’

    and the initial STUN phase is the problme

  53. jonas’

    moparisthebest, but with CGNATs, you can never know whether they always end up with the same egress IP for the same flow

  54. jonas’

    and CGNATs is where you need it most :(

  55. moparisthebest

    So we need our own stun protocol replacement with an auth secret ? :/

  56. jonas’

    and of course it requires a live channel between your xmppd and the stun server… which people wanted to avoid with REST mode

  57. jonas’

    moparisthebest, I don’t even know at this point what the attackers want to achieve

  58. jonas’

    so I refrain from making improvement suggestions at this ponit

  59. moparisthebest

    Or between your xmppd and the firewall in front of your stun server

  60. jonas’

    same thing really :)

  61. jonas’

    also I’m not sure how well that scales…

  62. moparisthebest

    So if you accidentally blocked stun for unfortunate cgnat people, they'd just use turn right?

  63. jonas’

    no, you need STUN before you TURN AFAIK

  64. moparisthebest

    If so, I think they'd have to do that anyway

  65. jonas’

    I need to dig deeper into ICE

  66. jonas’

    however, at this point, moving ports has mitigated the traffic 100%, so I’m not going to worry about that until the abuse alert triggers on the new port ;)

  67. moparisthebest

    It tries stun before turn before, the question is if no stun response will it go to turn

  68. moparisthebest

    Moving ports is at most a temporary fix, scanning the entire ipv4 internet is easy

  69. jonas’

    scanning IPv4 + all UDP ports is not that easy

  70. moparisthebest

    Likely shodan would already tell you about all stun ports

  71. jonas’

    if you need one day to scan the entire IPv4 range, you need 2^16 days for that on all UDP ports.

  72. jonas’

    even if we’re going to assume that only half of the port range is used and that you only need to scan every other port because of the alt port mechanic, that’s still 16kdays

  73. moparisthebest

    I guess it'd be easy to test what no stun, only turn, looks like in practice, just turn it off/block the port and see if calls go through

  74. moparisthebest

    https://github.com/robertdavidgraham/masscan can scan the entire internet in <5 minutes so I wouldn't count on numbers saving you here

  75. Holger

    moparisthebest, jonas', yes only the initial phase is the problem, a peer port is only allocated after successful TURN authentication (and can't be used for such an attack then either).

  76. Holger

    And yes you could avoid the issue by having some integration with the XMPP server.

  77. moparisthebest

    as far as scaling goes, ipset https://wiki.archlinux.org/index.php/Ipset can easily whitelist more IPs than your XMPP server can handle, doing this on not-linux is left as an exercise for the user

  78. jonas’

    ipset is dead tho? but I suppose nft has the same mechanism.

  79. Holger

    Dual stack foo will be fun again though.

  80. pepta.net

    > https://github.com/robertdavidgraham/masscan can scan the entire internet in <5 minutes so I wouldn't count on numbers saving you here Read full README before trying ;-) > Scanning the entire Internet is bad. For one thing, parts of the Internet react badly to being scanned. For another thing, some sites track scans and add you to a ban list, which will get you firewalled from useful parts of the Internet.

  81. jonas’

    moparisthebest, oh yes. challenge: a client is connected via IPv6, but its peer is connected via IPv4. Thus ICE can only succeed via v4, meaning that the STUN server needs to be reached over v4, but the v4 address of the client is not goign to be whitelisted

  82. Holger

    Yes that's what I mean (or vice-versa).

  83. jonas’

    moparisthebest, oh yes. challenge: a client is connected via IPv6, but its peer is connected via IPv4. Thus ICE can only succeed via v4, meaning that the STUN server needs to be reached over v4, but the v4 address of the client is not known by the xmppd and thus not allowlisted in the firewall.

  84. moparisthebest

    and in theory, it can't reach STUN and falls back to TURN, no problem

  85. jonas’

    moparisthebest, but then you’re using TURN which sucks.

  86. jonas’

    for no good reason, too

  87. jonas’ &

  88. moparisthebest

    the good reason is avoiding STUN amplification attacks I guess

  89. Holger

    It may well try to reach TURN via v4 and v6.

  90. Holger

    You don't want one of those to fail.

  91. moparisthebest

    you don't need to ip-whitelist the TURN ports

  92. Holger


  93. Holger

    There's no such thing as a TURN-only port.

  94. Holger

    Protocol-wise, TURN is a STUN extension.

  95. moparisthebest

    then you can't do this with iptables/firewall (unless it can tell the difference between STUN and TURN) and it needs done in the STUN/TURN server itself ?

  96. moparisthebest

    so that you can ip-whitelist STUN but allow-all with TURN ?

  97. Holger

    I understand nothing.

  98. Licaon_Kter

    moparisthebest: wherem do you get the ip's to whitelist exactly?

  99. Holger

    The entire idea only works if you handle STUN and TURN the same way. Regardless of whether you involce the STUN/TURN server or just the firewall.

  100. Licaon_Kter

    moparisthebest: where do you get the ip's to whitelist exactly?

  101. moparisthebest

    Licaon_Kter, the client IPs currently connected to your XMPP server

  102. Holger

    Right. I would think this could work (for STUN and TURN), modulo dual-stack foo (as usual).

  103. moparisthebest

    Holger, pseudo-code, in the turn server accept packet bit: if packet.is_stun() && packet.ip_is_whitelisted() { packet.respond(); } else if packet.is_turn() { packet.respond(); } // otherwise no response to packet

  104. Holger

    moparisthebest: As I said, that's not how it works.

  105. moparisthebest

    since stun isn't authenticated, you only respond if the IP is whitelisted, turn is authenticated so no need to whitelist there

  106. moparisthebest

    but it *can* work that way, no ?

  107. Holger

    That's not how it works.

  108. Holger


  109. moparisthebest

    why not ?

  110. Holger

    Only by changing the protocol.

  111. Holger

    If it worked that way, you could just enforce authentication for STUN as well.

  112. Holger

    It doesn't work because the attack happens in the pre-auth phase.

  113. Holger

    And authentication is challenge-response.

  114. Holger

    No way to avoid sending the challenge to the spoofed address.

  115. moparisthebest

    so the server talks first even in the auth/turn scenario? :/

  116. Holger

    And at this point there's no difference between STUN and TURN.

  117. Holger

    The server sends an "auth required, here's your challenge" in the auth/TURN scenario, yes.

  118. Holger

    The server *responds* with that "auth required, here's your challenge" packet to the initial client query.

  119. Holger

    But it's not obvious to me anyway how the idea depends on handling TURN in different ways.

  120. moparisthebest

    ugh, so yea, I guess it's: 1. fix STUN/TURN protocol 2. XEP so client sends all IPs it might use to the server after login for whitelisting :'(

  121. Holger

    2. alone would do the trick I guess.

  122. moparisthebest

    if the initial TURN packet from the client was "hi start a TURN session and this is my authentication key: stnhaoeunsth" that could be let through without whitelisting

  123. moparisthebest

    but, apparently that's not how it works

  124. pepta.net

    moparisthebest: AFAIK you cannot rely on ip src to be the same for two separate connexions because of CGNAT.

  125. Holger

    Right, if we had a way to negotiate that key outside of TURN.

  126. moparisthebest

    yep it's either 1 or 2, but 2 is the only thing I think the XMPP community alone can reasonable do ?

  127. moparisthebest

    pepta.net, yep jonas’ said as much, I don't have a solution there

  128. Holger


  129. moparisthebest

    what about... kinda port knocking ?

  130. moparisthebest

    you get a secret from the XMPP server, send it to the STUN port, and that IP is whitelisted for the real STUN request that comes later ?

  131. jonas’

    you can’t do (2) though

  132. jonas’

    a client doesn’t know it’s IP … it needs STUN for that ;D

  133. moparisthebest

    right, so you have a pre-STUN reverse-STUN that whitelists the client's IP for the real STUN ? :P

  134. Holger

    Ah indeed :-)

  135. jonas’


  136. pepta.net

    https://xmpp.org/extensions/inbox/sic.html needed ?

  137. jonas’

    moparisthebest, as I also said on-list… there are much more lucrative amplification attack targets than STUN servers out there.

  138. jonas’

    It makes absolutely no sense to abuse stun for that

  139. moparisthebest

    I don't think expecting sense from attackers will work very well

  140. jonas’

    oh I think it does

  141. jonas’

    and I think we must

  142. jonas’

    otherwise we might be fixing the wrong thing

  143. moparisthebest

    if we fix it so our STUN servers can't be used for amplification at all, it doesn't matter what the attackers do ?

  144. jonas’


  145. jonas’

    that’s not true

  146. jonas’

    ITYM can’t be used for reflection, in which case it might indeed be true

  147. jonas’

    because not all amplification fixes (such as requiring a minimum inbound payload of 100 bytes, e.g. padding) may fix other aspects of the attack

  148. moparisthebest

    I was just talking about adding authentication, so an authenticated XMPP client can do what they want

  149. jonas’

    we have that in theory… for TURN anyway (and leaving out the fact that the not authorized message is, again, larger than the request)

  150. Holger

    We don't.

  151. Holger

    I mean.

  152. jonas’

    we don’t?

  153. Holger

    The point would be accepting an auth token/password rather than doing challenge/response.

  154. Holger

    We do have 0215 to communicate a token to the client.

  155. jonas’

    isn’t that how it’s used?

  156. Holger

    But STUN/TURN doesn't have a way of checking that in the initial request.

  157. jonas’

    do I need to read the TURN RFC?

  158. moparisthebest

    so my hack proposal was a new auth'd protocol to whitelist the client's IP, and then proceed with regular STUN/TURN

  159. Holger

    I keep repeating myself on this point as well but people keep ignoring me until Philipp Hanck pops up 😂️

  160. jonas’

    I start regretting bringing this up

  161. Holger

    > he server sends an "auth required, here's your challenge" in the auth/TURN scenario, yes.

  162. Licaon_Kter

    moparisthebest: > the client IPs currently connected to your XMPP server Wait...but... > you also need to accept packets from their peers ...

  163. moparisthebest

    I mean literally something that listens for a single UDP packet with "auth my IP please, auth code: XXXX", and it only responds with "ok" if the auth code is correct

  164. moparisthebest

    then regular STUN/TURN takes over

  165. Holger

    Because back then people like Neustradamus explained how PLAIN auth is the end of the world and all we can allow is MD5-based challenge-response.

  166. Holger

    So now all we have to check a temporary token is MD5-based challenge-response.

  167. Holger

    moparisthebest, and you'd add that to what spec?

  168. octagon

    Can't we just rate limit the port with iptables?

  169. moparisthebest

    Holger, I guess a XEP specifies sending the auth code, and maybe the UDP packet ? :/

  170. croax

    octagon, would make it easy DOS?

  171. jonas’

    octagon, rate limiting, even by IP subnet, was surprisingly tricky. Tried that.

  172. jonas’

    not tricky to set up, but tricky to get right

  173. jonas’

    call attempts cause packet bursts, and if you have to retry within a given time frame because of an unrelated error, this will make it harder for you to succeed at having a call

  174. jonas’

    might be better if you also filter for specific STUN packet types, not sure if I went that far before moving on to changing the port

  175. jonas’

    moparisthebest, currently, my hypothesis is that we’re already golden if we change ports because reasons.

  176. jonas’

    even if the stun servers are rediscovered, they may not be useful for the type of attack

  177. moparisthebest

    changing ports is good, causes no harm, we should do it, I'm just uncomfortable with that being the final solution

  178. jonas’

    moparisthebest, I would do that and reconsider more drastic measures when the attacks come back

  179. Holger


  180. croax

    and you can randomize ports within a time-limited window, to enhance the proposed work-around.

  181. moparisthebest

    takes too long to actually do something else though, I'd rather start now

  182. jonas’

    croax, you can indeed

  183. Holger

    I totally forgot about the Short-Term Credential Mechanism (which is completely unused AFAIK): https://tools.ietf.org/html/rfc5389#section-10.1

  184. Holger

    But possibly precisely what we'd need.

  185. jonas’

    moparisthebest, yep if we have the time we should do the proper fix and go ahead and fix STUN/TURN themselves where necessary

  186. Holger

    I.e. my "all we have is challenge-response" claims were incorrect 🙂

  187. jonas’

    Holger, ehh, I was always assuming that that’s exactly what clients do?!

  188. Holger

    No they always do long-term.

  189. moparisthebest

    that does seem exactly what is needed Holger , what are the chances libwebrtc has support :/

  190. Holger

    Not sure e.g. libwebrtc will even implement short-term.

  191. Holger

    moparisthebest, right, that's the question.

  192. jonas’

    well, libwebrtc is on github, isn’t it?

  193. jonas’

    shouldn’t be that hard to implement support?

  194. jonas’

    add it to libnice and be golden

  195. moparisthebest

    it's google's

  196. jonas’

    worth trying I guess

  197. Holger

    Either way maybe that's the proper solution.

  198. moparisthebest

    if it was any other company, I'd be more optimistic

  199. Holger

    And then add to 0215 that we do short-term (unlike the rest of the world).

  200. Holger

    Well, good enough to recommend that.

  201. Holger

    Does Coturn support it?

  202. Holger

    (eturnal doesn't ...)

  203. jonas’

    I am happy to implement it there if it helps

  204. jonas’

    should be just a bit of C++ I guess

  205. Holger

    Just curious.

  206. moparisthebest

    "should be just a bit of C++" famous last words 🤣️

  207. jonas’

    Holger, it caims to support 5389

  208. Holger

    Well I claim that as well 😂️

  209. Holger

    But yes may well be supported of course.

  210. jonas’

    ``` -a, --lt-cred-mech Use long-term credentials mechanism (this one you need for WebRTC usage). ```

  211. jonas’

    so I guess the default is short term?

  212. jonas’

    I don’t set that though

  213. Holger

    Would sound weird to me but either way "WebRTC usage" is exactly what we're doing.

  214. Holger

    ``` 1/24/2015 Oleg Moskalenko <mom040267@gmail.com> Version 'Ardee West': - https admin server; - SSLv2 support cancelled (security concern fixed); - The server-side short-term credentials mechanism support cancelled; ```

  215. jonas’

    "oh no"

  216. Holger

    So maybe it's a simple `git revert` 🙂

  217. moparisthebest

    Revert revert!

  218. Holger

    ``` both modified: ChangeLog both modified: INSTALL both modified: README.turnserver both modified: examples/etc/turnserver.conf both modified: examples/var/db/turndb both modified: man/man1/turnserver.1 both modified: rpm/build.settings.sh both modified: rpm/turnserver.spec both modified: src/apps/relay/dbdrivers/dbd_mongo.c both modified: src/apps/relay/dbdrivers/dbd_mysql.c both modified: src/apps/relay/dbdrivers/dbd_pgsql.c both modified: src/apps/relay/dbdrivers/dbd_redis.c both modified: src/apps/relay/dbdrivers/dbd_sqlite.c both modified: src/apps/relay/dbdrivers/dbdriver.h both modified: src/apps/relay/mainrelay.c both modified: src/apps/relay/userdb.c both modified: src/apps/relay/userdb.h both modified: src/apps/uclient/mainuclient.c both modified: src/client/ns_turn_msg.h both modified: src/ns_turn_defs.h both modified: src/server/ns_turn_server.c both modified: turndb/schema.mongo.sh both modified: turndb/testmongosetup.sh ```

  219. Holger

    A simple `git revert` with a tiny bit of merge work.

  220. moparisthebest

    I wonder if that means they won't accept it upstream though

  221. Holger

    So I have a new secret plan how to sell my eturnal thingy.

  222. Holger

    Short-term-cred version will be expensive.

  223. moparisthebest

    Inb4 Holger hired by Google

  224. Holger

    One question is whether libraries such as libwebrtc support short-term creds, the other is whether they support authenticated STUN.

  225. moparisthebest

    you'd still need to violate that RFC a bit

  226. moparisthebest

    o If the message does not contain both a MESSAGE-INTEGRITY and a USERNAME attribute: * If the message is a request, the server MUST reject the request with an error response. This response MUST use an error code of 400 (Bad Request).

  227. moparisthebest

    you'd need to just not respond

  228. moparisthebest

    same with invalid username/password

  229. Holger

    Yes the TURN server would have to be configurable to work that way.

  230. moparisthebest

    or if the response was smaller it'd be ok but no idea if that's the case here

  231. Holger

    I still doubt it's about the response size 🙂

  232. moparisthebest

    yea you are right, it's less a problem if response is less, but a problem anyway

  233. croax

    As long as factor is <= 1 it would no more be a problem as attacker are supposed to only exploiting this to the benefit Of data amplification.

  234. croax

    As long as factor is <= 1 it would no more be a problem as attacker are supposed to only exploit this to the benefit of data amplification.

  235. moparisthebest

    croax, it stops data amplification, but still allows them to hide their true source IP

  236. croax

    Yet, their is Windows user for that. :-)

  237. croax

    Yet, their are Windows user for that. :-)

  238. croax

    But agree with you.

  239. croax

    Any UDP service without authentication would be vulnerable the same. Eg NTP ?

  240. croax

    Any UDP service without authentication would be vulnerable the same. Eg NTP, DNS, ... ?

  241. moparisthebest


  242. moparisthebest

    any that responds without authentication anyway, for example wireguard is not susceptible

  243. jonas’

    croax, NTP and DNS are prime examples of such offenders, yes

  244. jonas’

    moparisthebest, do you know how QUIC does things?

  245. moparisthebest

    jonas’: https://tools.ietf.org/html/draft-ietf-quic-transport-34#section-8.1

  246. moparisthebest

    So QUIC seems to think 3x amplification is ok...

  247. MattJ

    You should run it on a different port :)

  248. moparisthebest

    That is finally supported, though you can just ask DNS where that is