XMPP Service Operators - 2021-07-16

  1. ilmaisin


  2. ilmaisin

    i cannot for some reason open s2s connections from domain laiho.me to xmpp.org though i have properly set up certificates now

  3. ilmaisin

    from xmpp2.laiho.me i can federate just fine, even when it is hosted in the same instance

  4. ilmaisin

    I run ejabberd

  5. ilmaisin

    At ejabberd discussion forum they suggested that there could be a some sort of stuck connection

  6. Ellenor Malik


  7. zp1.net

    What error message do you get?

  8. zp1.net

    I run a prosody but i had something similar when my SRV entries in the DNS where buggy

  9. ilmaisin

    My srv entries are just fine

  10. ilmaisin

    > What error message do you get? I get connection timeouts

  11. ilmaisin

    Now i cannot join the conversation's muc either

  12. ilmaisin

    I am so going to nuke my xmpp server, i am fed up with this stuff

  13. mieum

    anyone have experience running an xmpp server behind a reverse proxy?

  14. mieum

    i've got prosody in an lxc container, with nginx directing streams on all xmpp ports to the container, and it works...but prosody is aware of the fact that the container and host IPs mismatch.

  15. mieum

    I've treid using the proxy_bind directive, but it breaks the stream >_<

  16. mieum

    maybe it doesn't matter so long as everything works?

  17. mieum

    seems like it *could* cause problems...

  18. MattJ

    ilmaisin: I'm an admin of xmpp.org, I can take a look at the logs later today and see if I can spot any issues connecting to your domain

  19. ilmaisin

    > ilmaisin: I'm an admin of xmpp.org, I can take a look at the logs later today and see if I can spot any issues connecting to your domain It would be great, thanks

  20. MattJ

    mieum: when you say "it is aware" - what do you mean? Is there a problem?

  21. mieum

    it works fine, but it thinks that dns isn't configured properly, beause the A record points to the IP of the host machine and not the container itself

  22. mieum

    I haven't figured out how to bind the stream to the host IP like you could with http headers. But, I'm not sure it matters...

  23. mieum

    so far it all works great haha

  24. mieum

    I am just unsure if there will be unforeseen issues because of it later

  25. MattJ

    If you're just worried about the output of prosodyctl check, don't be. There is expected to be an IP mismatch in this kind of setup

  26. mieum

    okay thanks MattJ :b I will rest assured haha

  27. MattJ

    But yes, Prosody won't know the real IP address of connections, and that brings some limitations

  28. mieum

    what sorts of limitations?

  29. MattJ

    E.g. don't try rate-limiting things that are based on IP address

  30. mieum

    ah i see

  31. Julian

    mieum: HAProxy developed a "Proxy Protocol" to deal with this problem. Nginx and Ejabberd have very good support for it. Maybe Prosody does support it as well.

  32. mieum

    Julian, I am using the proxy_protocol directive in nginx to listen on the xmpp ports.

  33. MattJ

    Prosody does support it, I wasn't aware nginx does... that's good to know :)

  34. mieum

    I've been using it with other non-http tcp streams, and it works really well! I was just alarmed by prosody's warning, which seems harmless like you said MattJ

  35. mieum

    maybe if I tinker more I can figure out how to properly bind the IPs in this case

  36. Julian

    Ok, maybe I misunderstood your problem. Anyway the limitations MattJ mentioned e.g. rate-limits per IP should be solved by using it.

  37. mieum

    Julian, are you using the proxy protocol that way with your server?

  38. mieum

    I'd be curious what solutions folks have come up with. I'm sure lots of people have been down this road before me :b

  39. Julian

    Yes, I do. I have nginx and ejabberd running on different hosts, connected via VPN. All incomming connections are accepted by nginx and passed to ejabberd using the proxy protocol.

  40. ilmaisin

    MattJ: also if you could try restarting the server to get rid of stuck connections? I have heard that prsosody sometimes has that kind of issues

  41. MattJ

    That was an issue that happened multiple years ago and was fixed

  42. MattJ

    Obviously I can't rule out a bug without checking, but I consider it unlikely and I don't plan to restart the server without reason

  43. MattJ

    I'll be at my laptop in an hour or so, then I can look into it

  44. mieum

    Julian, that's an interesting setup! are you using thr proxy_bind directive anywhere? is it irrelevant in this case?

  45. Holger

    MattJ: > That was an issue that happened multiple years ago and was fixed Oh I wasn't aware it's fixed. I felt it's not long ago this happened with muc.xmpp.org, and <https://issues.prosody.im/871> doesn't mention a fix?

  46. Julian

    mieum: I dont use proxy_bind anywhere. For my understanding the only usecase for this directive is as fallback if the backend doesnt support the proxy protocol. But I may be wrong about this.

  47. mieum

    Julian, that makes sense. For now I guess I will continue forwarding the streams with the proxy_protocol directive. So far I haven't encountered any actual problems

  48. mieum

    I just want anticipate future issues before too many people depend on the server

  49. mieum

    It's cool that it just works :) more people sgould host their own servers hehe

  50. MattJ

    Holger, I don't recall any recent reports of issues with muc.xmpp.org since we upgraded it. It was stuck on an old release for a long time due to an outdated OS, but it's all up to date now.

  51. MattJ

    And iirc the actual issue with xmpp.org back then was due to DNS resolution problems

  52. MattJ

    But my memory is hazy

  53. Holger

    I see.

  54. Holger

    I'll take note of the Prosody version whenever it looks like this issue again.

  55. MattJ

    That bug report was made against the 0.9 branch, we're now on 0.11, and many things have changed. We need to do some issue tracker cleanup :)

  56. Holger

    I've seen it every now and then until recently, but not sure with what servers/versions.

  57. MattJ

    There are a handful of things that could be responsible for the same symptoms, for example previous versions of Debian shipped with a libevent binding that had buggy timer handling (so timers to clean up dead streams may not trigger), for example

  58. MattJ

    If you actually do see it in the wild on an up to date server I'm interested to know about that

  59. MattJ

    Anyway, checking muc.xmpp.org logs, everything seems to be okay with the domain mentioned above...?

  60. MattJ

    It's actually doing XEP-0199 pings, and they are being answered

  61. MattJ

    ilmaisin, it looks like you moved servers and updated DNS, but there was still a connection open to the previous server and it was still answering for the old domain

  62. MattJ

    I just force-closed the connection and hopefully the next attempt will see the new DNS records and connect to your new server

  63. Licaon_Kter

    _It's always DNS_ :)

  64. MattJ

    Minor note, your DNS is technically against the rules in the SRV RFC, which states that the target of a SRV record should not be a CNAME

  65. MattJ

    Prosody, and probably most/all XMPP servers handle that anyway (it's a common mistake), but you may want to fix it anyway just in case something enforces the rules strictly

  66. ilmaisin

    Yeah, i'll take a look at it. Cloudflare's "cname flattening" will probably do the trick

  67. iiro

    MattJ: now it works, thanks alot!

  68. MattJ

    Excellent :)

  69. iiro

    Btw, why does xmpp stuff require certificates to be valid for the main domain too in this kind of configuration? I am not at all sure if it is net positive for security, since one compromising the xmpp server can compromise everything on the main domain too

  70. Holger

    iiro, actually only the actual service domains are required to be certified, not the host running the service.

  71. Holger

    iiro, otherwise you'd have to trust the DNS lookup.

  72. iiro

    Popularizing posh support for s2s would be nice

  73. Holger

    iiro, i.e. my client would look up your domain, the attacker would 'just' have to manage to return attacker.example.com for that lookup, and could then offer a valid cert for attacker.example.com.

  74. Holger

    iiro, the problem you mention could be solved by having service-specific certificates. Google and the rest of the browser lobby forbids large CAs to do that.

  75. Holger

    And as everything is HTTPS these days anyway, there's no lobby against that nonsense.

  76. iiro

    Yeah, but someone capable of editing dns can also get certificates

  77. Holger

    iiro, what.

  78. iiro

    Lets encrypt does dns validation

  79. Holger

    iiro, certificate validation is all about avoiding an attacker becoming MITM.

  80. Holger

    iiro, an attacker with write access on the network path between you and your peer, that is.

  81. Holger

    iiro, yes but attacking the way Let's Encrypt performs DNS validation is way, way harder than attacking your lookup in a lookup cafe WiFi.

  82. Holger

    iiro, yes but attacking the way Let's Encrypt performs DNS validation is way, way harder than attacking your lookup in a cafe WiFi.

  83. iiro

    Yeah, but i doubt anyone runs an xmpp server on a cafe wifi

  84. Holger

    People run clients in cafes.

  85. Holger

    If you're arguing that DNS traffic can't (easily) be forged in practice then I'd disagree. But if you were right, for what would we need certificates?

  86. iiro

    I am not really arguing, more like wondering. I just fear that current certificate infrastructure is over-reliant on dns anyway

  87. iiro

    Though doing the dns queries from many places might prevent certain kinds of attacks

  88. MattJ

    I agree - it's marginally better an verification than the old dialback protocol was (because Let's Encrypt queries from more places than a random XMPP server typically would), and they have some other protections

  89. MattJ

    As for which hostname to present, it makes sense to me that you need to present a certificate for the domain that you are claiming to be, regardless of your actual network address

  90. MattJ

    Unless a secure chain of trust from the service name to the network address can be established

  91. MattJ

    If I'm connecting to microsoft.com and DNS resolves to evil.example.com, I'd like to see a cert for microsoft.com, rather than evil.example.com to prove that I'm speaking to the right entity

  92. MattJ

    Unless I have some more secure way than unencrypted unsigned DNS to validate that microsoft.com really wants evil.example.com to handle their XMPP traffic for them

  93. MattJ

    FYI I closed that issue report: https://issues.prosody.im/871#comment-7

  94. iiro

    Perhaps there could be an exception that xmpp.example.com would be considered as good as example.com for xmpp purposes

  95. iiro

    But that too would require consensus

  96. croax

    POSH as is a nice and easy to set-up alternative solution but does not seem to be massively accepted at least regarding s2s. It also makes delegation easy for domains you don't own.

  97. MattJ

    Yeah, I think it's worth pushing POSH more. Also there are promising rumours about service certificates coming from the CAB forum, but we'll see if anything actually happens.

  98. croax

    Sure, hearing from this for a long time. Finger crossed :-)

  99. croax

    Sure, hearing from this for a long time. Fingers crossed :-)

  100. stpeter

    Wow, POSH. :-) I'll see Matt Miller at lunch today and mention this to him. :-)

  101. MattJ


  102. mimi89999

    Establishing a secure connection from lebihan.pl to tamytro.org failed. Certificate hash: 7ae362479e00b32a753eceb275fb93bf2a883399931f4598495d45c8099453a4. Error with certificate 0: certificate has expired.