XMPP Service Operators - 2021-07-16

Hello

i cannot for some reason open s2s connections from domain laiho.me to xmpp.org though i have properly set up certificates now

from xmpp2.laiho.me i can federate just fine, even when it is hosted in the same instance

I run ejabberd

At ejabberd discussion forum they suggested that there could be a some sort of stuck connection

ah

What error message do you get?

I run a prosody but i had something similar when my SRV entries in the DNS where buggy

My srv entries are just fine

> What error message do you get? I get connection timeouts

Now i cannot join the conversation's muc either

I am so going to nuke my xmpp server, i am fed up with this stuff

anyone have experience running an xmpp server behind a reverse proxy?

i've got prosody in an lxc container, with nginx directing streams on all xmpp ports to the container, and it works...but prosody is aware of the fact that the container and host IPs mismatch.

I've treid using the proxy_bind directive, but it breaks the stream >_<

maybe it doesn't matter so long as everything works?

seems like it *could* cause problems...

ilmaisin: I'm an admin of xmpp.org, I can take a look at the logs later today and see if I can spot any issues connecting to your domain

> ilmaisin: I'm an admin of xmpp.org, I can take a look at the logs later today and see if I can spot any issues connecting to your domain It would be great, thanks

mieum: when you say "it is aware" - what do you mean? Is there a problem?

it works fine, but it thinks that dns isn't configured properly, beause the A record points to the IP of the host machine and not the container itself

I haven't figured out how to bind the stream to the host IP like you could with http headers. But, I'm not sure it matters...

so far it all works great haha

I am just unsure if there will be unforeseen issues because of it later

If you're just worried about the output of prosodyctl check, don't be. There is expected to be an IP mismatch in this kind of setup

okay thanks MattJ :b I will rest assured haha

But yes, Prosody won't know the real IP address of connections, and that brings some limitations

what sorts of limitations?

E.g. don't try rate-limiting things that are based on IP address

ah i see

mieum: HAProxy developed a "Proxy Protocol" to deal with this problem. Nginx and Ejabberd have very good support for it. Maybe Prosody does support it as well.

Julian, I am using the proxy_protocol directive in nginx to listen on the xmpp ports.

Prosody does support it, I wasn't aware nginx does... that's good to know :)

I've been using it with other non-http tcp streams, and it works really well! I was just alarmed by prosody's warning, which seems harmless like you said MattJ

maybe if I tinker more I can figure out how to properly bind the IPs in this case

Ok, maybe I misunderstood your problem. Anyway the limitations MattJ mentioned e.g. rate-limits per IP should be solved by using it.

Julian, are you using the proxy protocol that way with your server?

I'd be curious what solutions folks have come up with. I'm sure lots of people have been down this road before me :b

Yes, I do. I have nginx and ejabberd running on different hosts, connected via VPN. All incomming connections are accepted by nginx and passed to ejabberd using the proxy protocol.

MattJ: also if you could try restarting the server to get rid of stuck connections? I have heard that prsosody sometimes has that kind of issues

That was an issue that happened multiple years ago and was fixed

Obviously I can't rule out a bug without checking, but I consider it unlikely and I don't plan to restart the server without reason

I'll be at my laptop in an hour or so, then I can look into it

Julian, that's an interesting setup! are you using thr proxy_bind directive anywhere? is it irrelevant in this case?

MattJ: > That was an issue that happened multiple years ago and was fixed Oh I wasn't aware it's fixed. I felt it's not long ago this happened with muc.xmpp.org, and <https://issues.prosody.im/871> doesn't mention a fix?

mieum: I dont use proxy_bind anywhere. For my understanding the only usecase for this directive is as fallback if the backend doesnt support the proxy protocol. But I may be wrong about this.

Julian, that makes sense. For now I guess I will continue forwarding the streams with the proxy_protocol directive. So far I haven't encountered any actual problems

I just want anticipate future issues before too many people depend on the server

It's cool that it just works :) more people sgould host their own servers hehe

Holger, I don't recall any recent reports of issues with muc.xmpp.org since we upgraded it. It was stuck on an old release for a long time due to an outdated OS, but it's all up to date now.

And iirc the actual issue with xmpp.org back then was due to DNS resolution problems

But my memory is hazy

I see.

I'll take note of the Prosody version whenever it looks like this issue again.

That bug report was made against the 0.9 branch, we're now on 0.11, and many things have changed. We need to do some issue tracker cleanup :)

I've seen it every now and then until recently, but not sure with what servers/versions.

There are a handful of things that could be responsible for the same symptoms, for example previous versions of Debian shipped with a libevent binding that had buggy timer handling (so timers to clean up dead streams may not trigger), for example

If you actually do see it in the wild on an up to date server I'm interested to know about that

Anyway, checking muc.xmpp.org logs, everything seems to be okay with the domain mentioned above...?

It's actually doing XEP-0199 pings, and they are being answered

ilmaisin, it looks like you moved servers and updated DNS, but there was still a connection open to the previous server and it was still answering for the old domain

I just force-closed the connection and hopefully the next attempt will see the new DNS records and connect to your new server

_It's always DNS_ :)

Minor note, your DNS is technically against the rules in the SRV RFC, which states that the target of a SRV record should not be a CNAME

Prosody, and probably most/all XMPP servers handle that anyway (it's a common mistake), but you may want to fix it anyway just in case something enforces the rules strictly

Yeah, i'll take a look at it. Cloudflare's "cname flattening" will probably do the trick

MattJ: now it works, thanks alot!

Excellent :)

Btw, why does xmpp stuff require certificates to be valid for the main domain too in this kind of configuration? I am not at all sure if it is net positive for security, since one compromising the xmpp server can compromise everything on the main domain too

iiro, actually only the actual service domains are required to be certified, not the host running the service.

iiro, otherwise you'd have to trust the DNS lookup.

Popularizing posh support for s2s would be nice

iiro, i.e. my client would look up your domain, the attacker would 'just' have to manage to return attacker.example.com for that lookup, and could then offer a valid cert for attacker.example.com.

iiro, the problem you mention could be solved by having service-specific certificates. Google and the rest of the browser lobby forbids large CAs to do that.

And as everything is HTTPS these days anyway, there's no lobby against that nonsense.

Yeah, but someone capable of editing dns can also get certificates

iiro, what.

Lets encrypt does dns validation

iiro, certificate validation is all about avoiding an attacker becoming MITM.

iiro, an attacker with write access on the network path between you and your peer, that is.

iiro, yes but attacking the way Let's Encrypt performs DNS validation is way, way harder than attacking your lookup in a cafe WiFi. ✏

Yeah, but i doubt anyone runs an xmpp server on a cafe wifi

People run clients in cafes.

If you're arguing that DNS traffic can't (easily) be forged in practice then I'd disagree. But if you were right, for what would we need certificates?

I am not really arguing, more like wondering. I just fear that current certificate infrastructure is over-reliant on dns anyway

Though doing the dns queries from many places might prevent certain kinds of attacks

I agree - it's marginally better an verification than the old dialback protocol was (because Let's Encrypt queries from more places than a random XMPP server typically would), and they have some other protections

As for which hostname to present, it makes sense to me that you need to present a certificate for the domain that you are claiming to be, regardless of your actual network address

Unless a secure chain of trust from the service name to the network address can be established

If I'm connecting to microsoft.com and DNS resolves to evil.example.com, I'd like to see a cert for microsoft.com, rather than evil.example.com to prove that I'm speaking to the right entity

Unless I have some more secure way than unencrypted unsigned DNS to validate that microsoft.com really wants evil.example.com to handle their XMPP traffic for them

FYI I closed that issue report: https://issues.prosody.im/871#comment-7

Perhaps there could be an exception that xmpp.example.com would be considered as good as example.com for xmpp purposes

But that too would require consensus

POSH as is a nice and easy to set-up alternative solution but does not seem to be massively accepted at least regarding s2s. It also makes delegation easy for domains you don't own.

Yeah, I think it's worth pushing POSH more. Also there are promising rumours about service certificates coming from the CAB forum, but we'll see if anything actually happens.

Sure, hearing from this for a long time. Fingers crossed :-) ✏

Wow, POSH. :-) I'll see Matt Miller at lunch today and mention this to him. :-)

:)

Establishing a secure connection from lebihan.pl to tamytro.org failed. Certificate hash: 7ae362479e00b32a753eceb275fb93bf2a883399931f4598495d45c8099453a4. Error with certificate 0: certificate has expired.