XSF Discussion - 2023-07-27

Has anyone worked on an approach that would allow you to terminate XMPP TLS on a load balancer, rather than on an XMPP server itself? With duplication of some certificate material, that's probably easily doable for BOSH and websockets - but StartTLS?

Only in my head.

IIRC the old stuff for nginx could do that

It's easy enough if you can cooperate.

https://github.com/robn/nginx-xmpp

Hard if you want channel binding other than the certificate one

or client cert auth

Like I say - those things are easy if you can cooperate (and trust). But yes, they're hard if you can't.

Guus, the xmpproxy thing by moparisthebest?

https://github.com/moparisthebest/xmpp-proxy/

Of course, I forgot (?) that can do starttls

Vaguely relevant new thing: https://www.rfc-editor.org/info/rfc9440

Oh, that's a lot more information than I anticipated receiving - thanks :)

Not using StartTLS but direct-tls / 5223 instead, that'd make things easier too, right?

Yes but no

No channel binding or cert auth, thus no s2s with Modern® servers :)

aye - I'm caveating my documentation that this largely applies to client-connections only (where SASL EXTERNAL is rarely used)

Channel binding is nice tho

the readme of xmpp-proxy suggests that it works for s2s, too

I wonder how that works (if it forces dialback or what)

Unless it uses the component client method

uhhh

that'd be nasty and awesome

I think I considered using that in the past

But since I would like Dialback to go away, of course it will come back and become mandatory again, just like vcard-temp :(

jonas’: iirc xmpp-proxy works for s2s by needing a module on the server side to accept cert auth information and such from it

this one? https://www.moparisthebest.com/mod_s2s_outgoing_proxy.lua

(taken from https://github.com/moparisthebest/xmpp-proxy/#how-do-i-adapt-my-running-prosody-config-to-use-this-instead )

I think so

Looks like that only overrides where outgoing connections are directed, in a way that will only work with older Prosody.

A little further down, there's another prosody script

It's all lua to me. ;)

singpolyma, that's outbound, not inbound

There is this reference in the text, suggesting that a module is being pushed to a more central repo: "Until prosody-modules is updated"

Just looks like things that override the security checks.

> But since I would like Dialback to go away, of course it will come back and become mandatory again, just like vcard-temp :( i assume with s2s mandatory tls recommendation, servers would just verify one another's certificates and that's the proof of a legit connection?

>Has anyone worked on an approach that would allow you to terminate XMPP TLS on a load balancer, rather than on an XMPP server itself? i have only ever terminated incoming connections but outgoing ones are still made via the jabberd

kind of the harsh reality of a monolithic network daemon

Yes xmpp-proxy validates client cert for incoming s2s connections (and supplies it for outgoing) and the prosody module exists to make it accept SASL EXTERNAL auth implicitly (because xmpp-proxy won't even connect to prosody unless it's already succeeded)

https://github.com/moparisthebest/xmpp-proxy/blob/master/contrib/prosody-modules/mod_secure_interfaces.lua this is the sasl external one

https://github.com/moparisthebest/xmpp-proxy/blob/master/contrib/prosody-modules/mod_s2s_outgoing_proxy.lua is the redirect connections one that also does it

Kev: Wouldn't it be reasonable to amend `Last Message Correction` to allow edits being targeted at the `<stanza-id>` in MUCs? I think it's a bit of a bookkeeping nightmare with with replies, retractions, reactions, and maybe future XEPs. I have submitted one issue in relation to reactions for dino* and I'll submit a 2 additional ones soon concerning replies in the context of edits and MAM. I suspect the dino devs are not the only ones who will get confused when they implement. Maybe with a namespace version bump? * https://github.com/dino/dino/issues/1468 ✏

I didn't realize I had linked ancient versions of the module from the readme :( thanks Guus

> Yes xmpp-proxy validates client cert for incoming s2s connections (and supplies it for outgoing) and the prosody module exists to make it accept SASL EXTERNAL auth implicitly (because xmpp-proxy won't even connect to prosody unless it's already succeeded) That means one should avoid having a way connecting to the server outside of xmpp-proxy, I assume?

Yes, at least with how this module is written

If you made it only do it on certain ports that would be an alternative

or maybe have something silly as a shared secret?

anyway, not the point. It's all pretty interesting. It's slightly veering off from my load balancing topic (assuming it doesn't do that), but interesting, nonetheless.

Right now it only connects to a single backend TCP port, but that could be made to randomly connect to one of a list easily, or that port could be load balanced

Obviosuly the web solution there is to add a proxy that load balances

Isn't srv records already ready to use for load balancing? Records with the same priority would balance per relative weight

The web shuns SRV records

Would be too elegant?

Only adding more and more proxies can be an acceptable solution on the web

i tried setting up a secondary prosody server using appropriate SRV records but i stopped when i realised that it might not even be a supported configuration yet (i'll have to check again though)

SRV records can be handy, but it doesn't play nice with HTTP-based solutions. Also, it is perceived as more static of a configuration than what some load balancers allow you to do.

slap xmpp-proxy in front and you can do both c2s and s2s over websockets :)

`client --[websocket]--> xmpp-proxy --[tcp]--> load-balancer --> xmppserver` ?

Guus, you need at least two HAproxy intsances and three nginx in that to advance to the next level

opal: doing it with SRV is sensible, but prosody specifically you need to be *very* careful using anything but a single node and for most uses it won't work or will only cause pain

yeah my idea was just to have the replica with read-only access to the database for auth, and that way it would serve as a limited-availability fallback where users could just not modify their roster or somesuch, which is acceptable

Guus: that'll work *now* yes, if you wanted naive load balancing in xmpp-proxy it'd be easy to add (just take a list of backend and randomly choose one to connect to)

if prosody can let me do that then i could set it up that way and have postgres replication as well

opal: you mean a limited-availibility fallback where users can't message other users on the same server and nothing else works, don't do it :P

you just lose MAM, right?

messaging is stateless after it's sent and as long as any plugins didn't hook into it

moparisthebest: I'm not explicitly looking for support. As I'm drafting Openfire documentation, I'm wondering about possible load balancing setups - this being possibly one of them - a bit of an outlier, maybe, but perhaps useful to someone.

omemo has a few extra steps involved but i'm not too worried about that either

i've run prosody with a downed postgres db, all it did was make sure i couldnt log in when i got disconnected

i could still talk fine

opal: but if you have prosody server A and B that both serve example.org and alice is on server A and bob on B they can't message each other, and outside servers will only be able to message one of them, it'll be a confusing mess

Guus: sure, still something for me to think about so thanks :)

moparisthebest: well, the idea is to only use B if A is down

moparisthebest, the intention is that server A is down, but yeah, i can see that scenario being unavoidable regardless (when server A comes back up and people don't disconnect from B)

singpolyma: sure but you can't do that with only srv records

so prosody would definitely need support for that :)

definitely a split-brain problem inherent in xmpp, idk if backup xmpp servers ever were implemented properly? with mail it's a no-brainer, since mail supports relaying fine

For the "a comes back up" case you could kick everyone off B manually. But yeah, edges abound that's why I said be careful ✏

mhm

If your server supports internal multinode coordination (such as ejabberd or jackal) then you can probably do this much more easily

ohh ejabberd handles this ok... i had weird luck with that before, but i could definitely look at how it's implemented there

"that" being "switching from prosody to ejabberd"

so i switched back :p

You need to consider what type of redundancy you want, because it greatly affects how you implement the clustering code. E.g. some servers are just quorum-based, and a node is either in quorum and available, or not (I think jackal probably works this way, as it's etcd behind the scenes, I think), other servers (e.g. M-Link) degrade down to single node operation with eventual consistency, so that e.g. two sites that lose connectivity can continue to operate locally.

"eventual consistency" seems like one of those things to introduce even more unpredictability to the end-user experience; i'd honestly rather have a "dumber" fallback in lower prio of SRV so at least people know when service is degraded (by receiving errors on certain operations)

Like I say - it depends what you want.

call me too careful but thats my thought process, especially having had to deal with other areas of xmpp where clients get confused on one server :p

yep

i hear "my client cant decide which avatar to show" too often

Although Openfire-specific, I'd love some feedback on the load balancing documentation that I've drafted in https://github.com/igniterealtime/Openfire/pull/2224 - if only to confirm that I'm not writing absolute untruths, or to highlight something that would be good to add.

oh Guus so openfire supports what we were talking about, nice

i'm giving it a read

https://github.com/igniterealtime/Openfire/pull/2224/files#diff-b9adf940bc94bbf75fb3c7174f963478a950e06f466e3e02cc3fb9709be9785bR60 superfluous apostrophe you cite your rfcs, srv priority example is right, the rest looks openfire-specific so you would know more about that than me, but yeah it all looks good

Thanks!

xmpp-proxy doesn't do BOSH (yet?) only starttls, direct TLS, websocket, quic

Otherwise looks good!

Thanks! No obvious omissions? It is hard to think of the thing that you're not thinking of...

(I removed the BOSH reference with xmpp-proxy)

merge it lol

additions will come

sure, nothing gets so much attention and maintenance as documentation! ;)

oh believe me theres always one person lurking like a hawk and then they email you when you already know about it

(im that person sometimes)

Happy to be on your list now :)

(it's merged)

nice

nicoco, about your corrections question, this would not work in MUCs without MAM. This also does not work in single chat. Why would you want a approach that works in all conditions (message-id) to change, where i suddenly have to consider the context

i find this way worse from a implementation standpoint

I compiled a table

XEP-0308 (Corrections) Message ID XEP-0333 (Marker) Message ID / Stanza ID XEP-0444 (Reactions) Message ID / Stanza ID XEP-0461 (Replies) Message ID / Origin ID / Stanza ID XEP-0424 (Retraction) Origin ID XEP-0425 (Moderation) Stanza ID

For Corrections, stanza-id makes not much sense

you can always correct only your own messages, so if we want to move a way from message-id to signal that unique id is in use, origin-id would be better

Doesn't message@id be problematic in MUC because the MUC is in no way required to preserve the ID attribute to all recipients of a message?

there is no reason to depend on server generated ids for the correction usecase

I thought that's the while reason replies etc talk about <stanza-id> ✏

I thought that's the whiole reason replies etc talk about <stanza-id> ✏

> Doesn't message@id be problematic in MUC because the MUC is in no way required to preserve the ID attribute to all recipients of a message? That was historically true, but isn't any more, unless I'm misremembering.

I thought that's the whole reason replies etc talk about <stanza-id> ✏

yes this was changed in the MUC XEP

MUC xep requires ID attribute preservation now?

and there is no relevant amount of servers in the wild that dont do this

So we can maybe remove the <stanza-id> stuff from other xeps also?

About what XEP are you talking exactly?

stanza-id was not only considered because of that problem with MUCs

there are other reasons depending on the XEP

Reactions, replies, retractions, moderation

did you see my table above?

There was only ever (to my knowledge) a single (not open-source) implementation that didn't preserve message@id through MUCs, and it's my understanding that even that implementation changed many years ago. origin-id was created as a workaround for that, but stanza-id was not.

retractions does not use stanza-id

moderation is a good use case for stanza-id, because i moderate a message in the server archive

its only MUC context

there is no moderation in single chat

MattJ: biboumi also does not I bet, but that's not super relevant for these xeps

<stanza-id/> used to be <archived by="server" id="0b359a68-2ca5-11ee-aa94-2f606f872410"/>

origin-id can (theoretically) just go away at this point. ✏

(in the MAM namespace)

But you still need other entities to be able to put an id on something going through, e.g. MAM.

But then it was suggested that we should make the ids generic, beyond MAM

Hence stanza-id

What Matt says :)

Kev, i would argue origin-id is good because a client can signal he understand the concept of unique ids and uses it

lovetox, might as well just say that every client using unique ids should put a certain prefix on the attribute value

We only have stanza-id because message@id isn't required to exist or be unique, basically. But some of these xeps could just say "if you implement this xep you need unique IDs" like corrections does

That saves another element and confusion between all the different ids

yeah if you design a protocol new

now it would be a breaking change, to save a few bytes

not worth it

> We only have stanza-id because message@id isn't required to exist or be unique

thats not true at all

the server needs to assign its own id

you cannot build a protocol where you hope that all clients use unique ids and insert this into your archive ..

How is that different from what I said?

The issue is that message@id may not be unique

you implied that if there was a standard that says message-id MUST be unique

message archives would need no stanza-id

what im getting at is the word "required"

it would make no difference if we write a standard and "require" it

because there is no way to verify uniquness

We could debate that edge but for now it doesn't matter since the spec doesn't even require message@id to be present so it's pretty moot

yes

But these newer xeps could require it

there are XEPs that reference your own messages, they should use origin-id or message-id (if backwards compatibility is needed)

there are XEPs that reference other peoples messages, and there in MUC its nice to use stanza-id because all messages are reflected by the server

But it would be nicer not to need to use stanza-id

yes

i think it would be best if they all switch to origin-id

its nice though that stanza-id cannot be forged

They already usually use message@id for 1:1 and then for muc say "or maybe stanza-id" I think maybe we can just remove that maybe stanza-id part...

i think in MUC, stanza-id, single context origin-id

with stanza-id you dont have to think about clients faking id

thats a big plus

Hmm. So the worry is that because you react by id only and not also sender what if a second sender does inb4 with same id? Hmm

and in single context, it makes no sense to fake id

FWIW I lean towards using the archive id for as much as possible, generally

in MUC

It's only a pain for the period of time where the sender does not know what it is

in single context it makes no sense

there is no gain in single context, only complexity

there is no attack vector

a contact that sabotages ids in single context, only sabotages itself

and you can simply block him, and stop talking to him

in MUC you can really get on peoples nerves and make chaos

I really hate that stanza-id exists at all, but I guess since we have it, now that this vector has been explained to me I understand the need better for these attachment in muc cases

Why do you hate that it exists?

You think message archives should be keyed by the ids chosen by third parties?

i think by accident we are in pretty good shape actually with the XEPs

all XEPs that reference other peoples messages, use Stanza ID in MUC

and origin-id / message-id in single chat

I think it might be okay to dislike some applications of it, but in my head it's <archived by="" id=""/>, and that's desired, but then it also happens to be a useful identifier for other things to reference

If I were starting from scratch yeah, sender chosen IDs with namespacing by sender. Basically like how we do with email

> all XEPs that reference other peoples messages, use Stanza ID in MUC Yes. This is just annoying for implemetation. But I understand why it is that way now ↺

and XEPs that reference your own message, its fine to use origin-id/message-id because receiver will validate the sender anyway

For stuff I send to 1:1 is something like self-carbons out there to discover the mam id?

No, but it has been on my radar for years as something that needs to exist

An alternative that was discussed at some summit was deterministic ids, which I do find attractive, but I think it has too many edge cases

Like mam id generated from message@id for sender only?

The suggestion was to do something like hash(stream_id + counter)

That way the client and server both agree (in theory) on what the id of each stanza would be

Could do mam id of. message@id + resource gets you the same thing with no possible desync

If client negotiates something useful about promising to send ids of course

That relies on the client sending unique ids

Server developers have a (deserved, IMHO) distrust of clients' ability to do that :)

Yes. So client would have to advertise such to the server

Well, if your client lies all it can do is mess up your own archive of your own sends

Server can also return error on duplicate id

but why consider this, if there is a possibility to make it not happen?

hash(streamid + counter) is easy enough

> hash(streamid + counter) is easy enough This sounds like a way to make everything always broken ↺

Also, why hash? Just streamid + counter but counters are very brittle

we use counters with streammanagement all the time

Yeah, and clients get them wrong all the time :)

(you can see this in the logs of most public servers)

the server can simply send an error if he calculates a different id

and reject the message

Different than what?

server is at counter 10

client sends counter 1

server rejects, because client cant count correctly

Where is client sending the counter?

no message is sent

yeah true, nobody said that

i assumed it

we use it as message-id

or something

Right. So just use message@id but only if it fits some rules we can quickly verify

Server returns error if you break the rules

Rules can be "no dupes" or "count right" either is fine by me. I like this general idea

Benefit to using counter (now we're getting crazy) server could use it as a SM assertion also

the pain with this is always, that clients need to support the old behavior anyway

but if you dont start you will never come to a place where you can drop it

sounds good enough, can be negotiated at stream start, was there any concerns, or just nobody moved forward with this?

For sure

oh it would be in the MAM Xep, so MattJ would be the author :D

There was a thread on the mailing list somewhere about it (which of course I can't link to right now)

I like this because it also strongly encourages good message@id at the expense of not being able to use uuid if we do the counter thing

Well, certain things that are currently deployed may break

For example, anything that encodes meaning into id attributes, which isn't commonly done, but there are definitely things that do

Oh yeah, I do all the time, but not from client

Also at least Prosody delegates choosing an archive id for a message to the storage backend, and this is in active use

message id was just an idea, but as the mam id is only relevant for ourself

there is no reason to not just simply add a new element where we communicate this

OR

I don't think new element helps anything here. Yet another element with the same content in it

hmm .. message-id is not only for yourself

its for other people

MAM id is only for yourself

But with this we can make them the same which is a big benefit

The issue is if as MattJ says you have a storage system that needs IDs to be a certain way (maybe wants them sortable or whatever) then this might not be an option

why?

the db does not need to use this id as its sort id

In some cases it might

it simply can apply a integer entity key to the row

Not everything is an SQL database :)

oh wait .... could this be our sortable mam id

but where is the problem, any streamid + counter is sortable

how can it be ever not sortable?

ah stream id changes on every connect

It's not just about being sortable, but that the server might encode information into the id

also while we're on the topic: stream id is traditionally considered sensitive

and without doing a full review of whether anything still depends on that, it ought to stay that way

then dont use the stream id, and simply communicate a mam context id in negotation

Sure

was there not a new standard about sortable UUIDs

take that + counter

but at this point why even use UUID

server can simply generate a sequential id on every connect

perfect, a id that is convertable to a integer

where is the downside to this, that solves so many problems

even the one i talked about a few days ago with keeping the sequence with backwards mam queries

ah .. no that was MUC

though MUC can simply use sortable IDs ..

... why would stream ID be sensitive?

I agree we can use some other thing (resource, new mam ctx id) if needed, but that seems like an odd thing to be sensitive

oh wait, this would not work at all, if we simply add a counter, it would only be sortable within our sent messages

not the received ones

singpolyma, it has at various times been used as a nonce-like value for input to various things, such as authentication

lovetox: correct

MattJ: ah, hmm. Ok

Alternative is to send down a message with same message@id and a stanza id child after mam recording. Or just be lazy and do full self carbons

this would also serve kind of as acknowledge that the message reached the server

like we have in MUC

I've been leaning towards self-carbons for some time

I tried to get it into the XEP before it was advanced (even as an optional feature), but it was generally opposed as the carbons protocol had been "stable" for some time

and then I lost momentum

Hum OK. Seems like, at least in groupchats, having everything relating to stanza ID when present makes more sense to me. I think I handle that correctly btw (happy to take reports if I don't), but > XEP-0461 (Replies) Message ID / Origin ID / Stanza ID In group chats, it's necessarily stanza-id according to the business rules. Are there cases where message id and origin-id are different?

There shouldn't be, but who knows

no, just cases where origin-id is not there

I'm 100% behind dropping origin-id everywhere

Whenever it is used you already have to deal with it being absent

yeah and? thats called a migration

with that argument we could never migrate to something new

A pointless migration, because you can just stay with @id

We just add an extra element for no reason

no, because it does not carry the semantic that the client understands unique ids

Why do you need that?

its just nice, to legacy clients, to not confuse them with stuff that will break them

If the client does not use unique ids, what changes?

except we say such thing does not exist anymore

then im fine with message id

i guess it only affects corrections

corrections is the oldes XEP of the bunch

with anything newer we could expect client to support unique ids

but yeah im fine with it, if we just say, fuck clients that dont use unique ids

Right

and in RFC6120bis we tighten up the wording for @id to make this clear

Then we only have two ids again, the one assigned by the client, and the one assigned by the server (and both are useful in different contexts)

XEP-0461 (Replies) XEP-0424 (Retraction) before they advance they would need to be changed, because the both use origin id

that reminds me to use message-id when implementing those xeps, as they should anyway always the same as origin-id

retraction particular makes not much sense if i think about it

its about retracting my own messages, why would anyone shoot himself in the foot and use non unique ids, and then implement retraction

It could be retracting a message sent from another client

But if that client is using non-unique ids, it also won't have origin-id anyway, so retraction still wouldn't work

Id references ought to have some scope scheme for specifying which origin the id was set by

MattJ, but that is exactly that case where it would be useful to know if the other client set a unique id

otherwise i retract 100 message instead of one by accident

i dont understand your argument that retraction would not work

if i have a worst case client that sets the same message id for every message, and i retract that, it will retract all messages

lovetox, do you have some specific reason to do it like that, rather than limiting retraction/whatever action to the last message with a particular id? ✏

In SQL terms, some `LIMIT 1` and `ORDER BY` clauses would solve it, or `break` out of the loop that does the update.

Punishing whoever were being silly by reusing IDs instead of doing something weird like affecting many messages seems sensible to me.

> in MUC you can really get on peoples nerves and make chaos after being on irc enough and being on networks / irc-ish protocols that dont support moderation, and also taking from nntp/usenet style moderation, i prefer better filtering tools over any sort of "group moderation"

but thats probably a cultural difference from how others want it

idk

sometimes people dont know what they want til they have it

soulseek is unmoderated too so people just ban/ignore whoever they want

in that vein, most xmpp clients have a tough time ignoring a user in a muc even though it's as simple as blocking the muc@example.org/nick

just isnt exposed properly in half of clients i used

opal: you want xmpp "killfile"?

pretty much

Zash, currently i JOIN the info if a message is retracted, so i dont think this is possible

i'd leave my own MUC unmoderated if i could just easily ignore anyone lol

even though it's a topical channel for my project

im too old-internet to care

opal, ignoring is a tool for a user, thats not moderation

you need to think of stuff like, that your server archive fills with spam, other sources that access your archive, like weblogs, where readers then cannot ignore something

not saying that its not a useful tool for a user, but i wouldnt say it does replace moderation

i have radical views on moderation i'll just save my breath

yeah i think both approaches are key here.

i do understand the point about archives being trashed but in my head i never trust a server more than any individual user

thats how i would design communication protocols, thats all

xmpp pre-dates me so i cant really say anything to change what we have now

web-of-trust style filtering never took off anywhere either cus thats complex to implement in such a way where it's also intuitive at a ux level

and, im not cocky enough to rush that and fill the gap yet

> web-of-trust style filtering never took off anywhere either cus thats complex to implement in such a way where it's also intuitive at a ux level Except Facebook ↺

oh yeah, the Algorithm

brb phoning up my buddy zuck

opal: how would you feel about a "local block" for muc that still shows the blocked person sent a message, but not what they said? So you know what people are reacting to, etc

sounds like what discord does; and i mean, tombstone stanzas can definitely be sent regardless, so the client can choose to show a message or hide it depending on what the user wants

when i ignore someone i personally want to pretend they dont exist

the question for the developer is, should i still save the message

yeah theres that too

(reminds me of weechat /filter)

would definitly make it a lot easier, but i guess most users would say: no

yeah a lot of users want a more hierarchical experience cus its what theyre used to and theyre able to trust people more than i can

and if no, then i have to design my whole application for that, constantly expecting that references to that message are received but its not existent and handle that gracefully

i guess we have to anyway

i have this block feature on my long list of things

years ago i was toying with creating a chat protocol and then just had total burnout dealing with anything related to designing a communication protocol lol

might toy with it later though

I personally find the tombstone UI less confusing, but yes it would also be slightly easier for me to build

> yeah a lot of users want a more hierarchical experience cus its what theyre used to and theyre able to trust people more than i can hierarchical and free for all are not the only choices though. a community can still make their space safe without having a single dictator or an oligarchy. ↺

and of course networks of consent and web of trust are another layer on that.