XSF Discussion - 2018-02-28

We do indeed

Hi, SàT support it too

SaT?

SaltyBones: https://salut-a-toi.org

Just read the XMPP Newsletter. Good work :)

Oh there is one already?

Is it available on a website? Or is it literally just a newsletter?

https://xmpp.org/2018/02/the-xmpp-newsletter-28-february-2018

neat :)

Basically it is a link list, but there were one or two articles that slipped my eyes

I love it how xmpp.is say they won't support the spam fighting manifesto, and then describe how they essentially implement each of the stated requirements <https://xmpp.is/2018/02/21/the-jabber-spam-fighting-manifesto/>

Ge0rG, infact xmpp.is is one of the servers I get spim hits from atm, *the hilarity ™️®️*

I wonder why

Maranda: Contact addresses for xmpp.is are https://xmpp.is/contact/ (support, admin, feedback, abuse)

please feel free to bother them.

🤣

When looking for xmpp.is in my log I found this one instead: https://isopres.de/impressum/

Thanks for the newsletter, by the way

Yeah, it's awesome!

I'd love to have the big ones (like EVE and Epic) mentioned on our twitter as well

this ID mess is a mess

Kev, there’s no reasonable way we can actually force clients to generate globally unique IDs though, is there?

so I don’t think that there’s an actual solution to the "appears to be rewriting history" issue.

No, I think there's not (I keep saying that, I think) - I wasn't arguing we can solve it, I was arguing that "they've got lots of power anyway" isn't the reason to not care.

oh, I must’ve misunderstood that

how would we be caring then?

I felt that "they've got lots of power anyway" was a "we shouldn't care". We should care, we just probably can't avoid it, so we carefully document it in security considerations etc.

It sounded like Simon was saying that malicious clients/servers are a problem not worth thinking about. I might have misinterpreted his words.

Yeah, that's not what I was trying to say...

Hm...how to sum this up...1. A client and server can claim that a message-ID was A when it was B; that is almost unsolvable but the other recipients just won't believe it anyway. 2. There is usually no authentication in XMPP so the point seems a bit moot.

the main issue is that an ID can be re-used

as far as I understand it

and since we use IDs as identifiers in various protocols (LMC, but also References and stuff), that’s a problem

Yeah, but there is a difference between assuming that it happens by accident and can be fixed and assuming that it is adversarial.

I’m confused

The only way to properly solve this is to limit the validity domain of IDs to a single session.

SaltyBones: "no authentication"?

I think one can reasonably argue with that statement.

Kev, I think they refer to "cryptographic authentication". A server can esaily forge a stanza for anyone to or from his domain.

Yeah, sorry.

Ge0rG, but then you could force the server to have some sort of UUID and a session counter and if you throw the three things together you get reasonable global IDs.

There's cryptographic authentication, even. It's just that it's applied to the domain, not beyond.

is there a licence for the Newsletter? Would be nice so it could be translated

goffi, has been discussed in the comm team channel

isn't all website content on xmpp.org under a single license

Should be, but I think that notice was lost at some point.

Kev, yes, but there is no message authentication so if you only store messages you cannot prove to anybody later that serverA sent you messageB with ID-C...

That's somewhat different.

Yes, just pointing it out because of your "somebody claims you liked a post about KKK" example ✏

SaltyBones, the issue is that that scenario can be caused by a client alone.

jonasw, the reason I ended up with this proposal is that it makes the situation better and it is very easy to implement.

a mailicous server is really powerful, indeed, and we generally assume that each user can trust their own server, and that they have to some extent trust a MUC service if they’re using one

SaltyBones: I think my issue is that the core of your proposal (the hashing thing) doesn't make anything better :)

Or, I don't see how it does.

Kev, that's a reasonable way to look at it. You could just as well not do the hashing and just use per-connectionserver-salt + connection-counter

That was just a fix for the "oh but the connection counter leaks stuff" problem...

Oh, no, you couldn't do counters, because of the leaks.

But as telling the server what ID to use for MAM isn't sensible, AFAICS, I don't think the server being able to predict the client ID buys much at all.

If the server can assure that the client ID is unique it can simply use that for MAM.

That's the main point... :)

SaltyBones, but that only solves the "client knows the eventual ID of the message"

it doesn’t solve any of the malleability things cross-domain.

that is only solved by your rewriting proposal, and I’m not confident that will work properly

except with a lot of complexity on the server

jonasw, not sure what you mean with malleability cross-domain ✏

SaltyBones: You're assuming that a server is happy to have arbitrary client-provided IDs as the primary query into the archive. I'm suggesting I don't think that's valid.

I think you have to let the server decide how it indexes its database.

Kev, they are not arbitrary at all because the server can verify that the client generated them correctly. Essentially they are just forced to use the same generation algo....

Of course if there are servers that don't like this algorithm for some reason than we should look at what that reason is and how to fix it. :)

SaltyBones, the reason for prosody/Zash is clear: their ID contains the date because that allows quick access to a bucket of MAM data

SaltyBones: and what was the conclusion of the discussion? Can we re-use? Tobias: I don't see any licence mention on the wesite, what is it?

I'm sure you can translate it when referencing back and mention the original author

goffi, translations would be appreciated and then linked to from the main newsletter. Not a real legal discussion.

Tobias: SaltyBones: I plan to translate in French to a popular website, but it require to specify licence, so I want to be sure I can set CC By-SA there. And yes I'll mention original post of course.

what's the SA?

Share Alike

goffi, join commteam@ and ask there.

SaltyBones: Tobias: asking there now, thanks

jonasw, they could just include a timestamp for timestamp queries and use an increasing salt to have an increasing index...

We can probably suss out all of these problems but I am not sure anybody but me is actually interested in doing that. xD

> but I am not sure anybody but me is actually interested in doing that. I know that feeling. Too well.

I'm glad to have this discussion. I'm not convinced that the proposal, and the complexity that goes with it, is solving any problems that a simpler solution doesn't.

SaltyBones, but timestamps aren’t monotonic

That is, it seems to me that the problems solved by this solution are the same solved by just saying to clients 'be unique in origin-id', and updating other XEPs to say 'reply with the origin-id' (LMC, Receipts, etc.).

yeah

SaltyBones: sorry, I didn't even read through the message-IDs thread due to lack of time.

Kev, and maybe adding a reply "this is your mam-ID" to client messages

jonasw, they aren't? :)

Or ignoring origin-id, using the stanza ID the way we always have, and killing MUC with fire :)

SaltyBones, clock corrections make it non-monotonic. and of course there’s the issue of clock sync between nodes

Kev: or just finally mandating that MUC must keep message IDs.

Or that.

And mandating that clients which want to employ LMC and other references must use sufficiently unique IDs

jonasw, ah I was thinking about the lamport kind of timestamp

that sounds complex

Maybe somebody wants to resurrect the Jul 2014 thread on MUC message IDs.

a little

jonasw, anyway if you have server generated timestamps and a monotonic counter the mapping should be pretty easy although not a NOP :)

You need more than monotonic, don't you?

SaltyBones: but clustering!

no wait, that was Dave's text.

but race conditions!1!

If you want clustering and query by timestamp I suppose you need the opposite of monotonic.

You want to merge archives by timestamp. Although that sounds dubious.

So, let's suppose that servers really want to generate their own ID for internal use. That seems fair but then why does the client need to know this ID for MAM? That's a bit fishy imho...

Because it's that ID that is the index into the archive.

I wonder if we can do XMPP over that link: http://www.vodafone.com/content/index/media/vodafone-group-releases/2018/vodafone-and-nokia-to-create-first-4g-network-on-moon.html

but why does the client need to know that

Because it queries the archive.

To do what?

I mean, there are obvious solution here as well: 1. The client query the server by its own ID and the server can try to use that as an additional index or 2. The server could just reply to client messages with the new ID.

But the whole situation is weird...what are the clients trying to achieve by querying the MAM?

There is some massive logical disconnect here.

You're asking why a user would want to retrieve messages from their message archive.

What's the point of having the archive if you *can't* query it?

But why would you need to know what s in the archive to query it

Have you read the XEP? :)

Well, I ve tried... :)

You say things like "Give me everything since message X", where X is the MAM ID.

Yes, but now we are asking the client to query by and ID which the server generated and didn't tell it about...

Why not just query by the clients ID or timestamp?

timestamps are unreliable

SaltyBones, because it is exact

Syncing on timestamp doesn't work, indeed.

timestamps are not exact

And the client ID means you're storing the primary index provided by the client, which enforces implementation details on the server that I'm not convinced we want to.

And maybe it's something we can live with, but I don't currently see what it buys us.

I know at least one implementation which can’t live with that :)

The message in question might be an incoming message, an outgoing message sent by that client, or an outgoing message sent by another client. You'd use the client ID in some or all these cases?

jonasw: If you mean Prosody's timestamp one, I think additional stuff's going to end up needed there anyway, for all the other things we were discussing at the summit.

But yes.

Holger: Well, you obviously can't in all, for id clashing reasons. At least, not with this suggested scheme.

That's why I'm asking.

So basically the client ID is not an option.

If you don't want to introduce another great mess.

Holger, that’s a very good point, I like it :)

another great mess! \o/

It is beautiful to see the first newsletter after XMPP Roundup and Jabber journal :)

I see a new redirection problem: http://wiki.jabber.org/index.php/....

I'm confused. Say the client re-logs in after loosing the connection and knows the MAM ID not just of the last incoming but also of the last outgoing message because we solved that somehow. He then queries MAM with after=$ID. How does he decide whether to specify the $ID of the last outgoing or the last incoming message? The ordering of incoming vs. outgoing messages on the client side might be different from the server side, no? (Maybe *this* can only be solved properly by having the server reflect IDs?)

yes, that can be solved by having the server reflect IDs

Holger: that's an awesome point

hah

that’s why I think we just want self-carbons by now.

It's the same point I made on the list earlier this morning.

yeah

But with more words ;)

Kev: Oh sorry, I didn't catch up yet.

The chat in here was triggered by me replying to the mailing list thread, I think.

yeah

Kev: except almost nobody read your mail, it seems

The ordering or messages in MAM can be different from the client?

Can we maybe somehow merge reflection of IDs with 0198 ACKs?

SaltyBones: yes

Holger: this is something I proposed when MAM first appeared.

Holger: I'd rather not, that's somewhat breaking layering.

Holger: 0198 and Carbons and MAM in a single unholy union.

Why can they be different and who is right? :)

Kev: Then the layers are wrong IMO.

Really?

Kev: I find it a bit embarrassing to define a protocol that has the server generate two responses to a single message.

198 is about the network layer and stuff getting through.

MAM is about the protocol layer.

Holger, two responses?

198 isn't per message

not even per stanza, indeed

jonasw: (1) ACK I got message with ID $count, (2) ACK I got the message with MAM $ID.

Holger, but the (1) ACK is explicitly requested by the client with an <{sm}r/>

Yes I'd usually request it per-stanza.

hm, I don’

*I don’t

So 198 could be bunching a load of stuff, for different stanzas (which may or may not be messages), and is generally going to happen once the server receives the stanzas, whereas 313 happens later, once it routes goes in the archive.

jonasw: If the client doesn't deem this necessary, why does it deem it necessary for MAM?

Ge0rG: Yay I'm good at re-inventing wheels.

Kev: some servers will only emit the 0198 ack after fully processing the stanza

Holger, because MAM contains things from other entities I suppose

yaxim will emit an <r/> after each message because mobile is unreliable

on the XEP-0198 stream, I know the order of the stanzas. In MAM, I don’t unless I get an in-order reflection of my own message.

If a client queries the MAM by last ID but the order in the MAM might be different I don't understand how it works. :)

Ge0rG: And if it was reliable you wouldn't need 0198. I never got the idea of requesting an ACK only every now and then.

Except for requesting it only once per bunch of stanzas you sent in one go, or so. In which case a single MAM ID response is fine as well.

Holger, when sending a bunch of messages at once, it makes sense to request an ack only after the last message

saves overhead

yeah

(or rather, sending a bunch of stanzas in general)

Yes what I don't get is how the requirements differ from those for MAM ID reflections.

Holger, hm, maybe that would work.

still requires some kind of knowledge about the relative ordering of messages you sent vs. messages you received and other resources sent

I don’t think we can get that without something on the stanza layer?

Why?

to be able to query correctly

The ordering is now defined by the order or stanza IDs you got on your incoming stream.

*the order of stanza IDs

how would I know when I send and receive a stanza at the same time and then my connection drops without stream management.

now I need to query the archive

which of the two IDs do I use to get a complete, dupfree picture?

You ditch unacknowledged messages locally and query MAM with after=$ID, where ID is the last ID you got from the server, no?

(I’m currently too hungry to think of a more sophisticated case where using the wrong ID would actually lead to *missed* messages, but there might be some)

okay, I need some food first

food for thought, if you will.

I'll try coffee.

And I'll read Kev's email :-)

MAM IDs in SM acks seems to be worth exploring

Depends whether you want to communicate to the client "this was the last entry in MAM at this point in the stream", or whether you want the client to know the ID of every message in the archive

Holger: except we need SM for IQs as well.

MattJ, last entry in archive should be sufficient for most cases

MattJ: what about giving back a list of message id / message ID pairs.

Ge0rG: Those will obviously not carry a MAM ID?

I'm guessing Ge0rG means a map of @id -> MAM-ID

I just love our nomenclature

What nomenclature?

Nobody can agree on what to call anything :)

Can't we just map jabber IDs to nonza IDs and be done?

:-)

This protocol is not for zimpies™

While at it, maybe just fix 0198 to return an ID for every stanza and ditch both <r/> and the counting which nobody gets right anyway :-P

Holger: but layers!

Ge0rG: The TCP layer is responsible for reliable message delivery.

Holger: no, TCP is a byte stream, not a message stream.

I think 198 is fine as-is, and I'm not keen on extending it

But yes, we do need to solve the MAM-ID-for-outgoing-messages problem

MattJ: from a smart-server dumb-client point of view, having four different mechanisms to track messages sucks.

So then we need a separate acknowledgment for MAM.

I'm talking of 0184, 0198, 0280 and 0313

Typically consensus has been about reflecting outgoing messages (in part, or in full), because this also has other benefits and we do it in Carbons anyway (just not for the originating resource)

MattJ: yes.

I wonder how that will play out with self-messages.

Heh

I understand where you guys are coming from, I just think this adds a bit embarrasement when you show the procotol to a newcomer for the first time.

Holger, and your preferred solution is?

Oh sorry, you already said

MattJ: Merging 0198 ACKs with 0313 message reflections.

Holger: XMPP is already mocked by HTTP developers. It can't get any worse.

But I see how just adding a stanza-id attribute and otherwise keeping 0198 as-is has downsides. So yes just adding a 0313 mechanism is probably an easier way forward.

FWIW, I think keeping 198 and MAM IDs separate is sane separation of concerns

For the most case I don't think we should be introducing newcomers to the protocol

It's just like other things where the end result is more convoluted than it would be if we addressed all this sync foo in one go.

It's a library problem, and the problem is most libraries just leave you with stanza building

with 0313 reflections we probably don't need to <r/> each message any more

We need new libraries!

We need more libraries!

There are only three(?) for python!

i need to port poezio to aioxmpp, then there’ll be only one *evil laughter*

jonasw: python-nbxmpp!

that does barely count as library

and sleekxmpp used to be a parent of slixmpp? There is also xmpppy

Ge0rG: Well what we really need is new library authors I guess, and at least those will have to be introduced to the protocol. In practice you'll have to understand most of that stuff as a serious client author as well, of course, even if your library is sane.

So we have five.

Holger: as it happens, the most active libraries are maintained by client devs.

See.

We have much NIH here.

I wonder why ;-)

So I'm not fully convinced that "but libraries!" is a good excuse for adding insanity to the protocol.

I can at least argue that I didn’t NIH, there simply wasn’t anything for python3-asyncio when I started. And I even considered porting sleekxmpp to asyncio, but I thought this to be not reasonably possible.

Holger, as both library and client author, I agree.

but I think keeping SM and MAM separate is sane.

jonasw: you are biased.

Ge0rG, why?

I also think that keeping SM and MAM separate is good.

And I argue in favor of a new type of session, which is MAM-Sub

yeah

that’d be great

> Still, I like the idea of MAM subscriptions as a replacement or augmentation for carbons Saying that for three years now.

Ge0rG, implement it pls

although bind2 will probably do pretty much that?

jonasw: bind2 is just a mechanism to carry things.

bind2 would have the effect of MAM-Sub though?

jonasw: nope

by doing MAM sync and carbon enablement in a single atomic step?

Let me make a strawman proposal of MAM-sub: - you initiate a bind2 session, supplying the last-known MAM-ID - the server doesn't deliver offline messages - the server delivers your pending MAM messages - the server auto-enables carbons and mam-reflections to you, starting to deliver everything after the MAM sync as "live"

yeah

so MAM-Sub is like Carbons but with mam-reflections

alternatively, the server could just give you the current last MAM ID so that you can do the sync asynchronously

while already receiving live messages

I'm sure I proposed that and a bunch of other nifty optimizations (0198 auto-resume/start in bind2) on the ML some time last year

yo

we just need implementations.

jonasw: processing MAM after live will be a pita, but okay.

depends on your client, I guess

I’d be fine with that.

has a considerable latency advantage, especially if you’ve been out for more than just a few hours

Ge0rG, do mam-reflections solve the issue Holger described between incoming and outgoing messages?

flow: yes.

flow: MAM reflections will be part of your MAM archive, right between incoming messages, properly ordered.

we just need to make sure that MAM-Reflections don’t rewrite IDs. this time for real *scnr*

Ge0rG, what, why?

wouldn’t you just have your outgoing messages in the MAM archive?

Ge0rG, so mam-reflections are done after the message has been added to your archive, both incoming and outgoing messages

flow: yes

jonasw: I'm not following

sounds good

you should write a XEP

or otherwhise the idea will possibly be burried in the standards@ archive

Ge0rG, why would you have the MAM reflection thing (presumably <message from="mam" to="your client"><forwarded><inner message from="your client"/></forwarded><stanza-id…/></message>) in the archive instead of just <inner message/>?

jonasw: wait, what?

what is a MAM reflection?

jonasw, I don't think that is what Ge0rg wanted to say with "will be part of your archive"

jonasw: whatever we make it to be

Ge0rG, I am super confused now

jonasw: could be a sent carbon of your outgoing message, or the outgoing message wrapped in MAM

yeah

okay

but WHY would you put that wrapped message into MAM again?

jonasw: or maybe just a small-ish ack with the MAM ID and your original @id

jonasw: I wouldn't

I don’t understand: 12:14:59 Ge0rG> flow: MAM reflections will be part of your MAM archive, right between incoming messages, properly ordered. this then

jonasw: I'd put the original message, obviously

jonasw: ignore it please

okay

then I didn’t say a thing since 12:15:01Z

jonasw: of course your *sent message* will be part of your MAM archive, plus the MAM-ID

yeah, that’s a good thing :)

Ge0rG, once MAMSub is active, clients will only receive messages not stored into mam via the usual way, all other archived messages will be mam-reflected, correct?

flow: wait, what?

Ge0rG, specific mam-reflections please

flow: no, you will receive all messages as usual, with MAM-IDs injected

I find our reasoning so far somewhat questionable. Because servers might want to use different IDs for messages these IDs should be reflected to the client so that it can make queries with that ID. Shouldn't a server simply be able to respond to a clients query if the client uses its original ID? Maybe this is not really practically possible anymore now but it seems somehow more logical. :) ✏

Ge0rG, and carbons still using forwarded?

flow: let me sort this out: you will receive all(*) incoming messages as regular messages, sent carbons from your other clients as sent carbons and MAM reflections of your outgoing messages as whatever works (e.g. forwarded)

all(*) = remember what I proposed at the summit / XMPP2 / routing2 brainstorming

But maybe routing2 is still too controversial

What gives you the impression that it is too controversial?

flow: it breaks existing XMPP routing

But it's opt-in. Do we have an example of a legacy protocol which breaks when another involved entity activated routing2?

flow: not that I am aware of. But the proble is that it changes semantics of bare/full JID routing, which is guaranteed to leak outside the XMPP2 domain

I guess that is just a fancy expression for "something could rely on the semantics and would break if someone else is using routing2"

The root problem is that an XMPP1 entity will happily send to the full JID of an XMPP2 entity and expect it to be treated in the XMPP1 way

Full-JID 'I'm xmpp2' annotation seems like it works though.

Kev: maybe, yeah.

Or heuristically 'fixing' xmpp1 full JID based on DPI, but that seems unappealing and probably fragile.

Kev, DPI?

Deep Packet Inspection.

deep packet inspection?

Which I'm abusing as a term to mean 'look inside the payloads'.

The problem with the annotation is that it feels like it's undermining the point of routing2 in the first place

If we're going to annotate, let's just annotate and we don't need to make any other changes

and that's basically hints, but with a stronger definition of how to process them

MattJ: Maybe, perhaps.

What else would you annotate, though?

Exactly - nothing

So XMPP2 is XMPP1 with annotations on stuff you want to treat as ephemeral

And changed routing rules for anything that's not annotated?

And clients need to know to ignore anything in an annotated message, because it shouldn't be getting them.

I don't know, I'm kinda concerned that people are going to go down the "Oh, let's annotated such-and-such special casing" like we have with no-copy and no-store at the moment.

Indeed

In principle, it's technically equivalent.

I still feel we might want to start sending messages from the bare JID instead of the full JID.

from or to?

From.

I hadn't considered changing from

You certainly want to be sending to the bare JID.

Although if we're saying "treat all messages to a full JID as to a bare JID unless they have an ephemeral annotation", that may be reduced, I guess.

Hmm conversations lost the message I sent here this morning from the backlog hm hm

I need to find time to write some specific words here, so we can bash them, I think.

https://marc.info/?l=openbsd-misc&m=151974573718360&w=2 - Alright, I'm not complaining about XMPP protocol design any more

Ge0rG, lol

Yup. OpenSSL. Still written by monkeys.

Random question without much thought, why can't the one true message id be implicit as the hash of the whole stanza?

It's hard to define what should go into the hash and then some "things" change those things anyway so the ID would change...

But isn't just hash the entire thing good enough?

moparisthebest, yesterday I would have said yes but now it is clear that people don't just want unique IDs they want very specific IDs and pick them themselves.

They can still do that

moparisthebest, just read the backlog from today :)

The public one is the hash, they can use whatever as the private one

Encoding implementation decisions into the protocol seems wrong

Especially when it has loads of downsides

moparisthebest, hashing XML is problematic, and in any case the same stanza may change en-route

Unless you're saying it's just hashed after the first hop

I was thinking about hash too, but the issue with hash is that you have to find one without collision. If you change, you'll break all existing ecosystem.

moparisthebest, the problem is that we leak the private one because it is required for MAM queries.

see my 13:25 comment :)

moparisthebest: Stanzas might change at every hop. So you can't just hash the whole thing.

moparisthebest [14:11]: > Encoding implementation decisions into the protocol seems wrong Hashing parts of a message into its id is just that

goffi: Hashes changing is a solved problem, at least, you just specify the hash used.

We could just replace messages with their cryptographic ids and become the next peer to peer distributed content storage network

Kev: yes, but what for already emitted IDs ?

goffi: They don't have an embedded scheme.

Ge0rG, I'm not saying hashing parts, that gets complicated, I'm saying hash the entire thing

as to changing at server hops, doesn't the id only matter between client and their server?

so my client sends a message which I know has id AAAA because that's the hash

my server can change it, if it does, it sends me back a message telling me AAAA is now BBBB

does that not solve everything?

No the id matters between endpoint entities. ✏

if any server can change the message, the id can only matter between a client and their server?

No, the id matters end to end.

But the content of a stanza might be changed at any hop.

So hashing to generate an id doesn't work.

so now you might have an id that is the same and different contents?

what's the point exactly?

See <delay/> for an obvious application.

what's the point in a@a.com having the same id as b@b.com ?

surely it only matters between a@a.com and a.com, and b@b.com and b.com ?

Because otherwise you can't reply to previous messages, which you obviously need to be able to do.

wait where is this concept of replying to a specific message?

as far as I'm aware, there is just a guaranteed order and that's it

In assorted XEPs.

LMC, Receipts, ...

You'd have to track the message id associations for all eternity.

it sounds flawed at a basic level, in a federated system, where a message can change at any server hop, how can you expect an id to refer to remotely the same thing on opposite servers?

but really hashing would still work right? the server knows the incoming hash and the outgoing hash

when it gets a read receipt etc etc, it just reverse maps it on the way out?

Ge0rG, you need to track that anyway because read receipts, right?

I mean the client has to do it not the server but still..

SaltyBones: the client won't know the effective id between server a and server b

and doesn't need to Ge0rG ?

moparisthebest: yes, you need to reverse map, for the lifetime of the message. Which might be months.

a@a.com can't talk directly to b.com

sure

moparisthebest: Message contents aren't unique so you can't use a hash as an ID.

inb4 nonce

Holger, I don't understand what you mean, I mean hash an entire stanza for an id

moparisthebest, you can’t reasonably expect a re-writing server to map between IDs for all eternity, *plus* to know all protocols where message IDs may be referenced

if you rewrite, you map, easy

shouldn't need to know any specific protocols?

moparisthebest: Entire stanza contents aren't unique either.

moparisthebest, a server would have to re-write a clients reference to another ID, e.g. for XEP-0184 (reciepts) or Last MEssage Correction.

Holger, I don't understand, if 2 stanzas hash to the same id they are the same stanza

moparisthebest: But they are still two stanzas.

moparisthebest, but maybe sent at a different time.

no they are just 1 stanza

like "that’s a good idea", I send that maybe ten times a week in here

a stanza is always its context

moparisthebest: Hah what?!

(aaand here we are at matrix’ DAG thing)

Do I have to mention what kind of hard fail hashes are? Does DKIM ring a bell?

storage-wise you store them once etc?

Maranda, yeah, good point

oh my god

moparisthebest: "etc" :-)

moparisthebest: What jonasw said.

ah you are saying if you send the exact same stanza every day or something, got it

but maybe the rest would work? if id's are only valid per-hop

and anything rewriting the id keeps a map for as long as needed?

moparisthebest, how long is "as needed"?

eternity?

well for a server it'd be as long as it had the message

especially with non-IM use cases I can imagine "as long as needed" can be quite a while

moparisthebest: So you're hashing contents and keeping a map because the contents of a given stanza may change and ignoring the fact that different stanzas might have identical stanzas. But apart from that hashing contents sounds like a perfect solution yes.

in mam/smacks/offline storage

moparisthebest, what about references to that message?

*identical contents

Holger, no I've scrapped hashing :P

Ah.

jonasw, surely those are only good while there is something to reference?

moparisthebest, another server might have a MAM for longer than you do

moparisthebest, good boy that's for the best 🤗

(scrapping hashes)

Maranda, to be fair that was my first question (if you want to uniquely identify a message why wouldn't hashing work?)

to which the answer was, we need to uniquely identify a message as well as it's position in the stream

There were many answers, and I think that's one of the few things that wasn't an answer.

jonasw, in which case it has a map?

I guess that was plentily answered already by multiple sources about the "why"

the answers I saw about hashing were 'well you have to decide what to hash' which is different

Beside that attempting to reinvent "a DKIM version" for stanzas/xmpp is a horrifying thought 😘

Maranda, but SPIM!!kk

dkim is an entirely different solution for an entirely different problem

that xmpp already has solved from day 1

(that problem is, is server X allowed to send messages for domain Y)

Can't we just store the message content on the blockchain and only exchange message IDs?

so what's the problem with a simple solution like, client sets id, if any server changes it it keeps a map, the end?

servers don't *have* to change it, but if they do, they keep a map

moparisthebest, sorry to contradict but that's not what DKIM ultimately *does*, not that it's important though.

that's the goal isn't it Maranda ?

Nope DKIM is more about message authentication, contraffaction and tampering prevention.

I think it has that side-effect because of the hashing, but the goal was server-that-sent-this-was-authorized-by-domain

You're confusing with SPF me thinks.. And both are failing that's why they had to invent a third, DMARC that somehow fails too.

and there are 2 methods, SPF doesn't hash but is only useable at first hop, and DKIM hashes and is therefore useable through multiple hops (as long as no servers change the hashed part :))

> DKIM provides a method for validating a domain name identity that is associated with a message through cryptographic authentication. http://dkim.org/

DMARC is not a 3rd, it's just enforcing/reporting on SPF and DKIM ?

still unsure why we are discussing this, XMPP already guarantees the sending server is authorized by the domain

Someone mentioned DKIM out of the blue, someone else responded :-)

I'd like to get back to FTP again!

DMARC, nope not just reporting, and I'd avoid reading just introductions. Saves headache laters.

I said it was enforcing and reporting

and that's all it is?

You did? Oh you did I'm blind apologies blame the cold exposure 😆

that's fully understandable, unfortunately :)

That was a grotesque example to explain how inadeguate I think hashes are in stanzas context, no big deal anyways. Brb.

Holger: Servers change IDs per-hop based on a hash of the contents, store the mapping between IDs, entities fetch the mapping over FTP and do the lookups there. Address of the FTP server is stored in a blockchain.

You're welcome.

I feel like you're missing an opportunity to use GOPHER in there someplace

Kev: Perfect!

Can we guarantee forward secrecy of ids?

Yes, but not perfectly.

Also, no need to use a hash. We could base64 the entire stanza into the id attribute.

No collisions, and no need for hash agility then.

Dave Cridland, ELOOP

We base64 the stanza including the original id attribute and then replace it with the result?

Without the id attribute, obviously. It'll solve the c14n problem.

Holger: No, you base64 it including the *new* id.

Ah. Now it makes sense to me.

Else the stanza's changing and you'd need a new id.

Kev, I don't know why you're being silly. My suggestion was practical, and just as sensible as all the others.

Dave Cridland, it’s possible to do

One of those two statements is true.

since there’s no length field, you can put the resul… nevermind

oh my god

I really should get to sleep

I just switched to this tab, I can't tell what's a joke and what's not any more

MattJ, assume everything is

MattJ: That's the joke.

and don't forget, today is opposite day :)

Tobias, OR IS IT!!?!??!!

So, to bring it back a bit. It seems this view is currently popular: The server needs to pick his own IDs, the client needs to use these IDs for MAM queries and we will add reflection so that using reflection and carbons a client gets copies of all its messages to have those IDs.

why is it more important for the server to pick it's own IDs than the client?

Concrete proposal: Set origin-ID uniquely on the client, add another id any time something archives it and wants it available, reflect that id back to the first client on first hop. Set the stanza id to the same as the origin-id, but generally ignore it and use the origin-id where available in things like LMC.

I don't think it is an issue of what is more important it's just that the server writers want to do that. I am not one so I cannot tell you why. :)

so I propose they can do whatever they want, they just need to keep a map?

moparisthebest: keeeping a map is a hard task.

moparisthebest: also actually replacing all references according to the map, because you don't know what's a reference and what's just random data.

I'm pretty sure it's one of the simplest tasks in computer science

Ge0rG, true but now we need the client to keep the map or to change the ID of old messages or something...it also seems hard :)

it's not naming, cache invalidation, or off-by-one errors :P

No, it's two of those things in one :)

moparisthebest: actually it is a part of cache invalidation.

Off by one naming of invalid caches

But the point stands, doesn't the work still have to be done but now it must be done by the client?

No.

It needs the origin-ID for references and the stanza-ID for MAM queries....

It doesn't need any mapping.

I thought the point was to hose all those and go to 1 id ?

and the message ID in case the origin-ID gets stripped

Ge0rG: I suggest (and it's a novel suggestion, because I know no-one's suggested it before) setting the id to the same as the origin id.

Kev: you'll never get the author convinced to do that

Kev, I'm not sure I agree that keeping two IDs for messages is very different from a mapping... :)

SaltyBones: the client needs to keep an index on the origin-ID, but probably not on the MAM ID

I need to look up messages by origin-ID for LMC and ACKs, but not when fetching an archive

Don't you have to merge the local and MAM archive somehow?

in fact I think keeping 2 id's and an index on one is the very definition of a map

Dave Cridland, I sent you an email with https://github.com/xsf/xeps/pull/593 for the council agenda :/

moparisthebest: the difference is that on the client, you store that as part of the message DB, and you can clean it up together with the messages

SaltyBones: These are logically two different things. You need (temporarily) a way of correlating messages and their replies ,which is origin-id. For MAM sync you only need a single ID, which is the latest thing to go in the archive that you've seen (and, depending on your model, even that is optional).

I don't see a situation in which you ever need to map between the two.

so what if the client sets an id, and if the server wants to change it, it just sends that info back to the client and doesn't keep a map?

what's the downside of a single id ?

moparisthebest: it won't work.

why not?

I'm sure this has been outlined in here multiple times today

client sends id=A, server sends back A is now B, client changes id=A to id=B, done?

moparisthebest: what if the client disconnects after sending A?

smacks/mam handles all that

moparisthebest: except when you don't know that A is B now and the smacks session expires

B still gets it with mam though

and C who knows nothing about id=A just ignores it

Kev, it seemed to be annoying for client devs to handle the different kinds of IDs and merge them and use the appropriate one everywhere but maybe I just misunderstood...

it seems to be annoying for both client and server devs

SaltyBones: I don't think it's a significant problem, just that sometimes people aren't sure which id to use.

(Because everything that talks about the id was written back when there was only the stanza id, and so didn't need to be explicit which one)

e) none of the above.

which turns out to be the significant problem

Yeah, maybe. From this far away I really can't tell how I would handle syncing local history with mam :)

There is only one id that MAM is concerned about, ignore everything else

Sure, so clients will store everything with origin-ID first, then add the stanza-ID when they receive the reflection and then just merge in other messages from MAM based on that, right?

I mean when everything gets this wrong you can pretend it's not a problem, just like everyone did for years with XHTML-IM, that doesn't make it not-a-problem though

SaltyBones, personally I don't really understand why origin-id exists

and to be clear I don't mean in the future things might get this wrong, I mean essentially any newly written thing gets this wrong now

MattJ: because of MUC reflections.

MattJ, I think most people agree that it is superfluous and should be the same as message-ID.

Ge0rG, and I think servers breaking MUC reflections are broken

and because back in 2014, somebody refused to mandate that MUCs shall retain the ID on reflection.

so lets do that now, because 99% of servers get this right

MattJ: OK. I'll revive the thread.

and stop making this id discussion so much more confusing

As I recall the most vocal opponent to mandating id preservation since changed their mind

Okay, the other reason was that on origin-id you can assume client-enforced uniqueness, which you can't on @id

what?

So then we're back to stanza-id for tracking between you and your server, and @id for end-to-end tracking

why would a client be able to generate unique origin-ID but not unique message-ID?

SaltyBones: a client isn't *guaranteed* to generate a unique @id

Who cares?

I don't know.

maybe people doing references.

The only entity concerned with the uniqueness of @id are clients that generate them

Reference them*

MattJ: clients that process incoming @id's too, for references and ACKs

so if you're doing receipts/LMC/etc. then just be sure you generate sensible ids

But if it *can* generate a unique origin-ID it can generate a unique message-ID, can't it? :) ✏

SaltyBones: yes. But with @id you don't know from the outside, with origin-id you do

Ge0rG, you mean it's not mandated in the spec?

SaltyBones: no. @id is optional

SaltyBones, that's the whole root of this discussion

SaltyBones: you could send all your messages with id="badgerbadger"

MattJ, maybe, I'm haven't been around long enough to have seen the root of this discussion. ;)

SaltyBones: https://mail.jabber.org/pipermail/standards/2014-July/028988.html

@id is optional and controlled entirely by the sending client, nobody except that client can use it for anything. If you generate an error reply, you reflect the id attribute, that's all

Ge0rG, ...I can't even imagine that "it seemed like a good idea at the time"

SaltyBones: MUC is full of such ideas.

Second problem is that some MUC server(s?) did not preserve the id attribute in MUC rooms, which broke some clients when they received their own messages back with a different id attribute

> Ge0rG> SaltyBones: you could send all your messages with id="badgerbadger"

I don't think this is true

at least for messages part of the same session

flow: > It is up to the originating entity whether the value of the 'id' attribute is unique only within its current stream or unique globally.

flow: now I could join this MUC with a multi-session nick, reset my session after each MUC message and you'd end up full of badgers.

or do the same to you via type=chat.

Right, but not within the same session/stream. I just wanted to clarify that

@id is an end-to-end property, so binding it to the session lifetime doesn't make any sense.

flow: technically you are correct.

just a matter of the definiton of end"

Winter's coming!

Nah, noh it ends

So, to sum up: Add reflection of messages to make it easier to figure out how to query MAM otherwise everything has to be the way it is...?

Maybe document better what every kind of ID is used for to make it easier for devs?

As far as I'm concerned there is one id set by the client and one id set by the server

<origin-id/> was invented mostly a workaround for the MUC reflection issue

MattJ, ...and the client set one is origin-ID the server set one is stanza-ID.

I strongly feel that origin-id should go

and that any broken MUC services should be fixed

flow: except it doesn't actually solve it, as seen with biboumi.

Okay, so move origin-ID to message-ID and forbid rewriting.

Ge0rG, m0ar pls

I'll bring it up on council.

flow, summary is that transports can't always preserve it

SaltyBones, why move? just use rfc6120 IDs like before

I mean conceptually, move the responsibility...

which puts them in the "broken MUC server" category as far as I'm concerned

FWIW, transports don't have to be written in such a way as they lose them.

Kev, what? parse_error ✏

The ids

Biboumi... 🤔

Kev, ah, you're saying that transports can always be written in such a way that they do not lose IDs?

not if it's impossible to keep a map of IDs >:)

SaltyBones: At the cost of complexity.

Sure, sure

Yeah forbidding ID mangling seems like it would be a very same thing to do

Should standza-IDs be transmitted btw? Or is that between client and server only?

s/same/sane

SaltyBones, the latter. It's stamped on incoming stanzas (to the user) by the server, to let the client know what id the server has assigned it (e.g. in the MAM archive)

and in some version of the future, it will also be stamped on reflected outgoing messages

And are stanza-IDs used anywhere else? They seem to be presented as this kind of general concept that can be used to add stable IDs for some sort of domain...

They started off as just a part of MAM. They were extracted out (probably wrongly).

Well, no, not quite wrongly.

Because they're used beyond MAM, they're also going to be used for Unread sync, and injected into Carbons and stuff.

SaltyBones, originally MAM used to stamp <archived by="archive-jid" id="unique-id-for-mam-queries" />

But the general feeling was that this unique id shouldn't be MAM-specific, but reusable

Yeah, I'm just wondering if it has actually ever been used for anything and if tha anything is also between client and server or not.

Yes, only between client and server (or client and client)

Unread sync is the obvious one that it's needed.