XSF Discussion - 2023-08-06

jcbrand, wanted to ping you about https://github.com/xsf/xeps/pull/1271 i added some notes after implementing, would be interested in a discussion and to bring this XEP to the finish line.

Also maybe the community can jime in on https://github.com/xsf/xeps/pull/1270, where the change would result in using origin-id and stanza-id in a XEP where both are not necessary at all.

I fear the only way for me to chime in is, to state that I really believe that we should always state the id *and* the id-assigning entity when referencing other stanzas. but judging from the past experience, that is (sadly) probably controversial

Namespaced IDs? I like it.

we should really try to figure out why we have such a hard time solving the "stanza reference" issue

Some sort of (jid, id) tuple would be good yes

Zash, not namespaced, qualified (by an entity)

flow, those concepts are very close in my brain :)

Zash, how about: <referenced-stanza xmlns='urn:xmpp:sid:0' id='xep359-stanza-id' by='muc.example.org'/>

Zash, granted, they are similar, and I couldn't define and state the exact difference right now

I agree completely about the reference. for messages too that would be nice.

awkward given many of the existing uses are attributes, fitting a whole tag in there would be weird, but changing the attribute would be awkward too

Doesn't <stanza-id> have a @by already tho?

Zash, it does have a by attribute

basically <reference-stanza> is mimicing <stanza-id>

ah, right, sensible

but I wondered if we should design it for multi-references

<referenced-stanzas xmlns=…><stanza-id …/><stanza-id …/></referenced-stanzas>

that would allow to re-use your stanza-id parser

the wrapper could plausibly live in each XEP that uses such a reference then?

Zash, and the advantage of doing so would be?

omg, solving problems nobody has

Zash, fwiw, <referenced-stanzas/> xmlns would be xep359 one

actually ignoring problems that are pressing and lead to hard implementation and less adoption

lovetox, let's say I would aggree: what would be wrong with the approach for the sake of consistency?

problem statement?

The lack of a by on replies, reactions, etc is basically the reason we have to have these rules and use <stanza-id> in muc etc

But otoh people seem fine with <stanza-id> so whatever. It's not a big deal I expect

nothing flow, absolutly nothing, only that it solves a problem nobody has

or do you really think when developers implement this XEP they care from which tag they parse the id attribute?

Zash, I assume that lovetox assumes that there is only one message ID in the MUC case, which can not be spoofed

flow: no, we always use <stanza-id> in muc specifically because of this lack of by attribute / spoofing problem for message@id

flow: the reason why we have hard times to do the referencing thing is that the referencing thing alone doesn't buy us anything.

singpolyma, ok, and when referencing a message in MUC you want to simply use the stanza-id@id but without the 'by' attribute? ✏

And then the referencing thing adds complexity to cases it might not be needed or lacks features in other cases ✏

flow: that is what is already done

the pressing problem for me is currently, that we have an influx of new XEPs that reference something, and people have no guideline which ID they should actually reference and *why*.

singpolyma, ok, and I see how I works in MUC cases (if you do what I assume you do). But I believe that we should simply be always include the 'by' attribute for consistency and symetry

Copy what an existing xep does?

lovetox: we actually have a pretty good rule for that, I don't see the problem

who do you mean with we?

and where is this rule written down

All the XEPs have it in a footnote

care to elaborate?

which does not seem to be the best place for a rule that appears highly relevant?

flow: if we used a by attribute we could get rid of <stanza-id> completely and just use message@id and get rid of the dependency on mam. But I think only I want that anyway so it's probably not worth it

singpolyma, if you get rid of mam, then what's your id provider?

flow, for many usecases you dont need an id provider, i guess you mean a server

by id provider, I mean the id assigning entity

is it the MUC service or the sending client?

or something else?

singpolyma: we do have clients out there that have shitty message@id generation, so no, we can't rely on it for features that need to work on messages generated by those clients

flow: sending client. As with correction xep

And as with 1:1 for other xeps

singpolyma, then which entity ensures that the IDs are not spoofable?

It's fine to say "you can't reply to messages of those shitty clients", but it's definitely not OK if we can't moderate them

larma: we already do everywhere but MUC.

singpolyma: we don't moderate outside MUC

flow, not sure about which spoofing you are talking, its only you your server and a contact in many cases

Oh sure, I agree moderation is different maybe because you're communicating with the MUC about its own archive. It's basically mam dependent even in concept

lovetox, MUC case: Alice sends message A1, Bob sends +1 reaction referencing A1

flow: that's what adding a by attribute to the reaction solves

lovetox, MUC case: Alice sends message A1, Mallory sends message A1, Bob sends +1 reaction referencing A1, but now it is unclear which message is +1'ed ✏

singpolyma, and in the anonymous MUC case?

flow, occupant-id

singpolyma: let's say we have a XEP for requesting a message with all its edits at once, how would the server know which edit to attach to if we relied in message@id and by attribute and the same message@id appears twice? ✏

singpolyma, but you are right, the by attribute helps here when referencing messages

seems you search for one thing that rules all

but you simply need to categories use cases

Mallory could just use Alice's nick while she is offline and thus have the same by ✏

larma: again, that sounds like specifically a mam extension so probably it would use mam IDs

user reatract use case, there does not need to be stanza ids, its unspoofable

singpolyma: wait, edits are a MAM extension now?

message correction use case, unspoofable

reactions use case, unspoofable

larma: fetching a list of things you said

singpolyma: sure, but that needs to be possible with the information the messages contain, we don't want to have a new XEP that extends the edit xep to be compatible with the new mam feature

larma, yes Mallory could also send while Alice is offline, but isn't that an argument to use MUC room MAM IDs when referencing messages? by using the MUC service as central arbiter for message IDs, all of those problems seem to go away. and modern MUC rooms want to provide MAM anways, then you get the MAM IDs for free ✏

Heh, you could even do <stanza-id> and pretend to have zero retention...

right

flow: I'm all in favor to use MUC stanza-id ID in MUC and this is the status quo as far as I can tell

Or at least should be

ok, because it appeared to me that we where talking about a MAM-less MUC future for a while. but I probably just misunderstood ✏

What we don't need to do is to state the by attribute because that's always given in MUCs (because only the MUC's stanza-id is ever referenced) ✏

by attribute is always unnecessary if we talk about XEPs where users send stuff to other users

servers tell us that anyway

For direct messages, by attribute could be relevant if we consider that a reference might point to messages of any of the two (not true for edits, but for example for reactions)

dont see how it is necessary in that case?

you reference a id, i attach the reaction there

What if you and me use the same id?

then i attach it to both

why would i care, its a reaction from another user, if he wants sabotage himself

so be it

its like the question, what if the client uses 1 as id for everything and makes then, a correction for message id 1

yeah .. then i correct 100 messages, not really a problem of the receiving client

I don't want to react with a thumbs up to your message "people are stupid", I want to thumbs-up my previous message "don't be stupid"

user will be annoyed to talk to other user who has a broken client, and will tell him ✏

larma, i see your point, if i think about a enviroment where all kind of clients use non-stable ids

you are right, in that case namespacing it with by is necessary

but we are talking about 1:1 case

I agree from that standpoint that intentionally misbehaving clients are not that much of an issue because it only breaks the conversation of those two users and that conversation is already broken if one side breaks rules in any other way, so that's something acceptable to me

its evident that the client is broken, users will sort it out themself

in MUC case stanza id is used, and its definitly unique, so nothing of that sort can happen

lovetox: what is non-stable id?

ah sorry, meant non-unique

Ah, yeah, that's why Dino for example won't allow you to reply to 1:1 messages without origin-id, because origin-id explicitly requires uniqueness and suggests fully randomness

yeah, and i feel thats unncessary

but i respect your motivation to support older clients

i actually would want to have stats which clients are really in the eco system that do this

the main problem for me is currently origin-id/message-id

it actually stops me from implementing more stuff

because its uncertain if some people will change it in the future, and i have to deal with this later

and larma, i bet you allow message corrections without origin-id

so quite inconsistent for me

message correction is a rather old XEP

and it works only really if the client has unique ids

Why? The ruleset is pretty easy: if origin-id is present on 1:1 messges use it for references. If not use message@id for references. Then maybe decide to not allow message@id values in references that seem not unique/random ✏

its not about the rule

its about if people decide to change the rule in 2 years

i want to have a stable enviroment where i can depend on that design choices by XEP authors / board are thought trough and not changed suddenly

What could be a possible change of the rule that would break with that rule

as i said, there are other people, that dont have your opinion that origin-id is something we need

and it can go away

I totally agree we can get rid of origin-id eventually (except for MUC reflection usecase)

so you can think of something that breaks that rule

:D

I also don't want it, but shitty clients are out there and we have to handle them somehow

look i dont care which camp wins

lovetox: no, if we drop origin-id you use message@id as per the rule

thats a bad rule sorry, there is no XEP in stable that mandates origin-id

is it too much to ask to force a decision now

The only things origin-id is effectively being used for is: - let others know your message@id is unique - be able to identify if a message@id was modified

I hope we see ourselfs not as a community divided into camps that need to win, but a community striving for consensus

instead of doing nothing and let authors further publish XEPs until we are in a position where it needs to be supported for all eternity

you know for my implementation its not trivial to simply switch to another id

You know that origin-id and message@id are the same on literally every implementation that has origin-id at all?

yes, thats why i think about to just ignore it, and simply always reference the message id

but this is a risky implementation

*every implementation you are aware of

as it seems not clear which way things go with ids

Sure, let's change the origin-id XEP to require that

So that we can be sure any future implementation also has this feature that origin-id = message@id

would be a good start

wouldn't that make origin-id obsolete?

flow, some people want the info if a id is unique

flow: we still need it for the two things I mentioned above

(which wouldn't be a bad thing, I suppose)

> The only things origin-id is effectively being used for is: > - let others know your message@id is unique > - be able to identify if a message@id was modified

What happens if I write a server-side plugin that strips origin-id if == message@id ?

larma, the second case is also non-existent

lovetox: MUC reflection

if you think there about the MUC case

and the XEP was ammended

larma, are you aware of a implementation that is not 20 years old that does this actively

?

then lets fix it

every new implementation will be compliant with the XEP

What do you mean has been amended?

MUC #stable-id is fully optional

of course because MUC is a old stable xep

Biboumi still does not support it afaik

because it splits message right?

Yes

damn

ok ignore what i said

and what does it do then?

Tangent to that, XEP wishlist: advertising certain message limits

does it invent new ids?

or uses the same

i wonder does biboumi need to do this?

only because it splits the outgoing messages, why does it need to split them for the reflection to xmpp?

but on the other side, non of these XEPs work for IRC anyway

retraction, reaction, moderation, whatever

and stop, you can simply use the muc feature as indication for that

dont need to have this info on every message

so its actually only the 1. case

and there i would argue, its not worth it, in MUC we use anyway stanza id, so origin-id is not necessary

and in 1:1 case its simply not important, people just migrate away from broken clients

also your message reply use case, i dont see it

you only allow a message reply if the other party uses origin-id

but if it does not use origin-id, it will also not have support for message replies

so it will not see it anyway

nothing happens here

your case would only be for clients who dont have unique ids, but support the most new XEPs like message replies

in 1:1 it should be more feasible to rely on disco#info too, but it may still be weird given multiple devices

no its simply not important for 1:1

There might be two messages with the same ID in a 1:1 chat and still multiple Clients that understand replies

It's not like 1:1 chats only have two clients

yes there can be cases constructed where something breaks, but its not worth it to add something new for that

It's not something new

larma, we have message corrections since years

i heard no complaints that its totally broken because there are clients with non unique ids

Message corrections only work on the last message, so the id doesn't really matter anyway

no, thats implementation decision

and we will definitly support this for not only the last message

What most clients do is to just edit the last message with that id if there are duplicates

Which is reasonable in the case where the XEP explicitly says it's only to be used for the last message

But I might want to reply to the older of two messages with the same id

ok so someone uses a old client, and a very new client at the same time

Like many users

and you need to reply to a message from the old client

its very far fetched

That have Pidgin on Desktop and Conversations on mobile

and for that you mandate that all clients forever add a origin-id tag

That's the two most popular clients

How is that far fetched?

cost/benefit is not there for me sorry

For the most popular desktop client

Pidgin uses maybe a non-unique id

but that you hit the same id multiple times in a single chat with a contact

is still very low probably

at what point do you conisder the cost of the measure?

if somebody can construct a single case where something would break, you add a mitigation not looking at the costs?

even if i would agree, that there can be constructed a case with a used client, that happens to 1 in 1000 messages, i would still not mandate every client in existence forever, to add a origin id tag

No, we surely should come up with rules as to when message@id is fine even if it's not an origin-id. See my previous message. ✏

> The ruleset is pretty easy: if origin-id is present on 1:1 messges use it for references. If not use message@id for references. Then maybe decide to not allow message@id values in references that seem not unique/random

I would propose to accept any uuidv4 message@id the same way as origin-id

Do any of these debates have bearing on what servers will need to index for fastening-like functionality?

So the only thing we need to do is to mandate that origin-id must match message@id if present. Then it's fine for a client to only rely on message@id but require it to be uuidv4 to do replies/reactions/...

MattJ: for MUCs it will always be the MUC assigned stanza-id, I think everyone agrees on that

For 1:1 it's more complicated because your index must be on (local-user's-jid+remote-user's-jid+(origin-id/message@id))

But that's basically status quo

😔

i think its wrong to do all this complexity to support unmaintained client like pidgin

(Because uniqueness can't be guaranteed on 1:1 IDs and thus must be restricted to the chat)

What complexity? Requiring that origin-id must match message@id? Or requiring that message@id must be UUIDV4 for references? ✏

I'm in favour

the complexity in code, in my database design, when i join tables on IDs and must conisder this

UUIDv7 would also be nice

For me those two rules are to reduce complexity from what we theoretically have right now.

no, what we could have is easy, use always message id in single chat, always stanza id in MUC

> the complexity in code, in my database design, when i join tables on IDs and must conisder this Don't implement origin-id. If we have those two rules you're good to go without it.

yes thanks, thats why i want a decision

Database design is its own field for a reason.

something that shows me, this is the future i can implement that

not just words :)

You will still need to take the participants into considerations when doing database joins

Sorry, typing on an annoying phone keyboard. I'm in favour of anything that moves us back away from origin-id and towards a sensible place.

The @id is the origin id, and it's regrettable that the semantics weren't more watertight in the RFCs

Because I shouldn't be able to react to messages in your chat with Matt just because I discovered the ID of a message in that chat ✏

of course larma :) its just you dont want to have OR in your join clause, message_id = id OR origin_id = id

from what i read its very inefficient and a full table scan

and you're assuming an SQL database anyway

> UUIDv7 would also be nice Agreed ;)

lovetox, what if you have your own internal ID, along with a lookup table for message-id → gajim-id ?

That's basically what we do in Prosody

of course, a third column :D yes in code everything is possible, im not saying this is an unsolveable problem

but having a proper decision, and not needing to consider all possible futures makes it easier to choose the best way

So we all agree that we have to make flow adjust XEP-0359 to: - Ask specifically for UUID v4 or v7 (instead of current just any UUID) - Mandate that origin-id, if present, must match message@id if the origin is the sender of the message. ?

I just wish there was a more compact UUID representation

yes that would be a good start larma

Zash: isn't that up for the underlying transport/storage engine to handle ✏

altough i have no opinion about the specific version of UUID, could it not be enough to say that implementors must choose a fitting UUID themself

larma, I'm thinking for wire protocol

larma, I'd be in favour of that move

lovetox: UUID v1 doesn't entail any randomness

lovetox, since you often speak of using timestamps, surely you want UUIDv7 for everything, being basically a timestamp + some random noise

hmm no, i want a sequential number in a archive

and timestamp generated by uuid7 certainly is somehow dependent on the computer time on the sender

i currently would have no use for this

Sequential counters, the easiest thing in computer science

larma, but would it not be better to say, choose a uuid which has some randomness

instead of mandating a specific version

Zash, or do you mean that the archive uses uuid7 as stanza id

could be interesting yes

the question is, is uuid7 sortable?

Zash are you sure it includes randomness?

i doubt it

> The UUIDv7 format is designed to encode a Unix timestamp with arbitrary sub-second precision.

ok seems they all gurantee global uniquness

so they need to have somekind of randomness in it, how this still can be sortable is beyond me

ah i get it, they put the timestamp in front

and hope that no entry has the same timestamp

yeah so that would work if a archive simply uses that as stanza-id

its a bit dependent on the correct time on the machine, but we have that anyway, as the timestamp of the message is recorded by the server

so if it is wrong, it will be totally broken anyway

Even with unsynced clocks, you get things that are near in time closer in the index, which might be good. Or maybe it doesn't matter, I'm not a DBA 🤷️

Unordered things like UUIDv4 seem unfriendly to indexes

Tho I assume that proper databases can handle that too

but this uuid7 think is mainly useful for some kind of distributed databases, where i would need some global assignment of id otherwise

for a single machine storage, simply use a autoincrementing field which all databases have

That's what all UUIDs are designed for

no need for complicated uuid7

*Universally* Unique IDs

Unique forever and troughout the entire universe!

yes but archives dont need globaly unique ids

Pretty sure this is actually insane overkill

ejabberd does this btw

Especially if IDs are scoped per JID

it simply uses a timestamp as ID

( JID, 64-bit number ) or so would probably be fine

Relevant to the above discussion: https://github.com/ulid/spec And https://en.wikipedia.org/wiki/Snowflake_ID is worth a look.