XSF Discussion - 2019-03-24

MattJ / flow : https://github.com/xsf/xeps/pull/771

Would something go horribly, horribly wrong, if a server simply adds a stable/unique stanza ID to any message that it processes?

It makes client life hard/impossible, sadly

Clients would no longer know when a message is archived or not

So they don't know if the id can/should be stored for later querying (e.g. for catch-up)

I do not like the fact that we're deducing that something is archived by merely detecting the presence of something that's supposed to be an opaque identifier.

(also, I don't have a better suggestion)

Guus: that's why <archived> existed

What's the impossibility for clients, exactly?

Hmm I don't quite see this problem. There's no guarantee archived messages will remain in the archive forever anyway.

And I don't quite see how the info whether a locally stored message is also in the archive helps the client.

In my book it's fine to add a stanza ID to all messages. It may actually help with non-MAM use cases.

It'd make my implementation a lot easier...

> Would something go horribly, horribly wrong, if a server simply adds a stable/unique stanza ID to any message that it processes? In short that's the basis of our XEP that we use to ensure message delivery. Works well.

Andrew Nenakhov the basis is that something goes wrong if we do (and you found an alternative), or: you do it, and you've seen that nothing goes wrong?

Client sends stanza with provisional id, server stamps it with 0359 unique and stable id, sends this id to client as a confirmation.

Guus, define wrong )

Mushroom-clouds on the horizon.

We centralize everything to work via server archive. If archive breaks, kaput, yes

> Mushroom-clouds on the horizon. In case of an all scale nuclear weapon attack, im protocols will hardly matter anymore

Andrew Nenakhov I do like to make sure though that IM protocol design decisions do not cause an all scale nuclear weapon attack.

Holger, you're right (there is no guarantee archived messages will remain in the archive forever)

But if it's not in the archive, the client can assume it was purged and it needs to re-fetch the archive

MattJ: how can it fetch a purged archive?

Guus, I mean in the sense that old messages are purged

Mind you that it is Sunday, I'm an idiot, and did not have enough coffee

I don't understand.

Messages in the archive are not kept forever on most deployments

The oldest messages are removed after some expiry time (let's say 30 days)

with you so far.

but why does the client need to re-fetch the archive if that happens?

need/would want to

So a client that wants to receive all messages works by continuously remembering the id of the last archived message it received

When it goes offline for a couple of days, it will come back online and request all messages since the last id it saw

if anything, it'd do a massive amount of data transfer only to end up with _less_ local history?

With me still?

yes

So now it goes offline for two months

The last id it saw is no longer in the server's archive

So it performs the query and gets item-not-found

so it knows that the message has been expired, and any messages in the archive are messages it has never seen before

because they are all newer

right. And if we'd slap on a stanza-id on every message without archiving, it'd _always_ get a item-not-found, assume its local cache is older than that's on the server, and it'd download all history, every time.

that's what you're saying, right?

Yes

I'm guessing that it'd not do this if the server doesn't advertise the MAM feature though.

Sure

I feel like the whole "item-not-found means get full archive" thing is a hack. A server could lose a message for other reasons, e.g., storage failure causing recent stuff to be lost, or deletion of specific message due to gdpr, or some bug, etc.

waqas, it's not allowed to

> and it'd download all history, every time. It would download up to the date it just requested

waqas I was trying to formulate a similar remark in my head.

waqas, it can replace with placeholders if it needs to

Which may or may not be the whole history

MattJ: Storage failure isn't something someone can't be allowed to have.

waqas, handling storage failure in defined ways is entirely sensible

sure, but soleely depening on 'item-not-found' based on a last-known ID still seems ... hackish...

sure, but solelely depening on 'item-not-found' based on a last-known ID still seems ... hackish...

Guus, it's defined by the XEP to be this way, it's absolutely not a hack

I mean, what else would you guys propose??

MattJ: Not really. If you lose a disk and restore from recent'ish backup, you'll have a situation where supposedly every recent message would be item-not-found..

MattJ it's a lot easier to disagree with stuff without having to suggest better alternatives 😉

waqas, you can't just rewind time like that in most systems without consequences

Yes, and given that laws of physics disallow "messages can't be removed from archive after acked", a protocol shouldn't rely on that.

what if the client asks for the last-known ID archived by the server?

MattJ: To be clear, I think a sane recommendation would be if item-not-found, get archive by some timestamp based setup, but trying to get archive from beginning of time is silly in such a case.

(What I said above?)

(removed bad idea)

Yep, listen to pep.

Yes, but the server was relocated to a different timezone and the admin forgot to set it to UTC

Dates don't include TZs? :s

Almost all popular dbs people use (mysql, postgres) in their default replica settings, when the master node is lost and another takes over (or a restoration from backup happens) will potentially lose recent writes. If the MAM XEP wants to assume that wouldn't happen, I'd consider it pretty silly.

waqas, if you want to write your own XEP go ahead

MattJ: Do you see the problem I'm pointing out?

Maybe 'silly' isn't the best classifier here.

> Yes, but the server was relocated to a different timezone and the admin forgot to set it to UTC do we need the XEP to account for this?

Guus, do we need the XEP to account for any of this?

Well, if we can modify it somehow to be more resilient against data corruption, and allow for easier re-use of stanza-id, I think it'd add considerable value.

waqas, I don't think a server that can't provide a durable store should be able to claim it does

There's a simple fix for this, the XEP already has a flag to tell the client that the results are not necessarily persisted

MattJ: I'm asserting that the vast majority of MAM deployments can't guarantee durability in a disk-lost scenario. Recent writes being lost is a fact of life, you can't spec your way around it without mandating things you have no way to mandate.

I look forward to your PR

Note that I don't think the MAM XEP has to change, just the assumption that item-not-found always means MAM storage was deleted up to that item is wrong.

So yet another hidden thing for client devs to think about

In a world where fsync doesn't necessarily mean data was durably stored, and SQL dbs multi-master replication defaults to async mode (and is rarely used anyway), that's reality.

MattJ where in the XEP is the what I called 'hack' described?

I was looking to see if the exact wording would make me think of hints for improval

Guus, it quite possibly isn't

ah ok.

I'd love to be able to add stanza-id's everywhere, without implying that this means that MAM is available.

but doesn't service discovery sufficiently guard against that?

Adding stanza-id doesn't imply MAM is available

Buf it MAM is available, it implies you can't put stanza-id on every stanza

I'd like be able to. Is a feasible solution one that allows the client to request the id of the most-recent MAM entry, in order to verify if it has that one in its local archive?

If the XEP doesn't currently define the 'store the id of the last message, assuming that it is the last ID in your server-sided archive', there might be room for a change like that?

Guus, one of the main premises of the XEP is history sync, this would break it

Forget the message purging issue for the moment

If the client records the id of the last message it received, and then later uses this to query an archive, what would you propose it do if the id it happened to remember wasn't an archived one?

item-not-found

and then what?

Naively (I'm not client builder): I'd see up until what date I'd have a local archive, and retrieve from there.

So fetch by timestamp?

with some wiggle-room, but yes.

That way you'll either get duplicates or miss messages

And that's not hackish?

Duplicates I can de-dupe with the message ID

We could have just built the whole XEP on timestamps instead of ids if we're happy with that

misses would be bad.

It's an ugly hack

well, let's not rewrite everything just yet - I'm fairly certain you've put way more thoughts into this than I have 🙂

This is not something I would accept a rewrite for, for certain

The correct fix is to re-introduce a way for the client to know whether the message is in the server's archive or not

> Buf it MAM is available, it implies you can't put stanza-id on every stanza Depends on server implementation, no? The server just must be able to respond to the before/after requests.

so, why can't it ask for the last-recorded message id in the archive?

Guus, how does that help?

what's my last message? do I have this? no: resync everything.

Guus, that's broken

Just because you don't have the last message in the archive doesn't mean you don't have the first

E.g. ejabberd uses timestamps as IDs, so it doesn't matter whether the queried ID is archived, before/after still does the right thing.

resync everything from the last one that you have, I mean.

Guus, you don't know what the last one you have is

Holger, multiple stanzas with the same timestamp?

Microsecond accuracy, if you hit that in practice then yes it breaks.

Holger, what about clock drift then?

MattJ how don't you know what your last message is? You can order your local archive chronologically, use the last one?

I'm not against using timestamps *in* the id, but it's wrong to use them as the id with no extra logic

Guus, the last what? I don't know which ones the server archived

MattJ: Clock drift across cluster nodes? That would break as well yes.

Hmm, my parents just walked in. Wife is preparing for 'the stare' again.

(Or do you mean clock jumping backwards? That can't happen with our clock except server restarts.)

Mattj, but if archiving is enabled, you can assume that the messages that you have in ... aah, I don't have the time to further discuss this now, sorry.

('stare')

I'd love to pick this up later.

Holger, using the system's monotonic clock? or something custom?

got to go now

Erlang has a thing that doesn't jump back, not sure how it's implemented.

Anyway yes this is not the most robust solution against such pathological cases of course (it just has other nice properties). Whatever I just wanted to say that MAM doesn't imply only archives messages have an ID per se.

(ejabberd doesn't actually add IDs to non-archived messages, though I keep pondering with it.)

Holger, as discussed, things will break (read: get hard/impossible) for clients if you add stanza-id to non-archived stanzas

which is not a good situation, and should be fixed

Maybe I misunderstood the breakage vector. I would've thought things will be fine as long as the server is aware how the non-archived IDs are ordered compared to the archived messages.

MattJ, just in case you're interested, this sounds like custom clock that (attempts to) adjusts towards OS clock by changing frequency (up to 1%) while avoiding jumps: http://erlang.org/doc/apps/erts/time_correction.html#No_Time_Warp_Mode

Fun

(At the cost of risking incorrect offsets of course, so they warn against doing this.)

Holger, the server knowing how to interpret the ids is not really relevant... unless you're saying it should not return item-not-found but quietly accept ids that don't actually exist in the archive

That would cause weirdness with clients that try to fill holes

and probably other stuff

> unless you're saying it should not return item-not-found but quietly accept ids that don't actually exist in the archive Ah yes that's what I'm saying. IIRC 0059 suggests doing just that (wasn't it even a SHOULD?).

But I'm on my phone and the sun is shining. Gonna shut up now 🙂

Is it too late to start over with MAM?

Not using RSM for a start

Trying to use existing building blocks has just caused confusion and unintended consequences

Well MAM is still experimental :-°

What about another bump?

Everyone would love that

That's a thing I don't like in general. The XEP is still experimental but in reality it's just as if it was almost Final. If you change anything everybody is going to grump

It certainly still has open issues, as a spec

Sure. I'm not just talking about MAM, that's how I feel about our specs in general

Can't have it both ways

Just this morning it was mentioned that XEP-0313 being Experimental is a reason Pidgin doesn't have support

I'd say that's an issue with developer expectations. If you implement it as experimental, know that it's likely going to change

(And even more, really, draft, even a final spec can be amended with another spec, so..)

MattJ, isn't that just an excuse from pidgin devs? :p

It's not a viewpoint I share, but I'm biased

Are devs expected to implement experimental xeps?

If a standard explicitly has a big red warning at the top, and warning or no warning is subject to radical change... if I had a limited amount of free time, would I want to implement it?

"While implementation of an Experimental protocol is encouraged in order to determine the feasibility of the proposed solution, it is not recommended for such implementations to be included in the primary release for a software product (as opposed to an experimental branch)." — https://xmpp.org/extensions/xep-0001.html#states-Experimental

waqas, in the meantime, it's a needed feature

And it's even in the compliance suite..

That's the real problem (that experimental or not, it's a needed feature)

I'd say both these criteria (needed feature / compliance suite) put even more pressure on the XEP to go to draft/final. I'm not saying I like it

And as you say there are still areas that need to be improved

Maybe there should be a rule that compliance suites can't recommend draft specs. In the hope that people focus/provide feedback on XEPs that are needed

I thought there was

Well if there was, MAM shouldn't be in there

nor carbons? (last call ended but it's still proposed)

We should stop calling them "compliance" suites

MattJ, I am not sure if using existing building blocks caused confusion. It appears to me that not clarifying how they are intended to use and are allowed to use (think for example if <before/> and <after/> can be used in the same query) is causing confusion

flow: they can't, the end :)

That is what I would also say, but it is at least underspecified in XEP-RSM