jdev - 2025-03-02

lovetox, is the backloading only for messages before your first join?

Mostly yes

Because I'm not aware of any client being able to fix "mam holes"

For example, Conversations loads the ~most recent~ 750 messages that follow the last message you know of ✏

That means you'll end up having an "mam hole" if there were more than 750 messages.

I'm guessing that downloading all messages more often causes undesired resource contention than that it addresses a user-perceived problem? How many users will notice missing messages further back than the last 750?

maybe for search functionality, but that could either be done server-sided (if not e2e-encrypted), or the client could refresh the entire archive when an end-user performs the search.

It's not the last 750. It's the 750 messages following the last message your client knows about. So, if a 1000 messages were sent when you were offline. You'd be missing the current context.

If you fetch only the most recent messages (for example the past day), can the client tell that there is an "mam hole" in the middle, and fix the hole when I scroll up to the position of the hole?

It's possible to keep track of those holes; the question is whether clients choose to - in most cases they don't as it's deemed uninteresting ("most users don't care")

lissine: as you say conversations specifically doesn't get a hole but rather fails to fetch up to current instead. I guess once you receive a live message after that it creates a hole. I submitted a patch to fetch current instead thus creating a hole but generally they don't take outside patches often

we should simply add a new feature for sortable stanza-ids

then this problem would be gone

or rather it would be easier to implement

Yeah, I would love that. I implemented it for prosody once already but of course we need to standardize a feature to advertise it

what do you mean once? is it gone again?

Well it wasn't accepted and I've been told there are other implementations around also

lovetox, let's say we have the following scenario: you didn't open Gajim for a few days. Gajim synced only the last day of history. Now, if you scroll up to the position of the "hole", how would Gajim be able to tell that there's a hole there?

sorry, sortable stanza ids do not fix the hole problem, if you can call it a problem

it fixes the restore order of messages from database problem

> it fixes the restore order of messages from database problem But that's not a problem currently? ↺

yes it is

clients just dont care, and it seems to lead to no big problems right now, apart maybe from time to time some bot messages in wrong order

if you request history always forward, from your last known message, then the insert order into the database is also the order you load the messages

but if you begin later to fill holes, or do backloading

then the insert order into the database is not anymore the order you can load the messages

if you then sort on timestamp, then you will not be able to find out the correct order if 2 messages have the same timestamp

that two messages have the same timestamp becomes very unlikely if servers have milisecond resolution

hence you could sort on timestamp

but thats a bit ugly, better would be the server has a sortable stanza-id

i think dino has a "fill the hole" implementation if im not wrong

In practice, when two messages were sent in the same second, it usually means they were sent concurrently (i.e. one is not a reply for the other) So the disorder doesn't matter much

you are right, if you exclude bot messages

or any automatic generated message

if we only had users, you could argue no human will send 2 messages in the same second

at least not while normal chatting

so as said, lots of edge cases, all in normal chat operation probably rare

thats why you could say, lets get the 95% solution for now, and live with the 5% where it goes wrong

but on the other hand, if a server can request a timestamp from the system, and write it to the database, i see no reason why it cannot generate a sortable stanza-id

We can also produce a sortable ID locally but it's a lot more work and I'll admit I keep putting it off ✏

how would that work in a multi device world

or it probably works, if all clients have perfect syncronized time?

but thats unrealistic

It wouldn't be based on time but based on order in the mam archive

please expand

how can a client know the order in the MAM archive

Because mam always delivers the items in order and you know what item you asked for before/after of

(or for live message they are delivered in order and always append)

ah ok i understand

yeah we talked about this, i started with that, but aborted because i lost faith

its probably shorter i learn erlang and provide a sortable stanza-id impl to the server

and that work would benefit the whole eco system and not just my client

It's gonna sound awkward after this time, but I've built gloox against opessl - it works with starttls now

Maybe should report a bug to your distro then that it doesn't work as built

> its probably shorter i learn erlang and provide a sortable stanza-id impl to the server Isn't there a famous saying about this? Something like: > Every problem in XMPP can be solved by adding another ID, except the problem of too many IDs ↺

> its probably shorter i learn erlang and provide a sortable stanza-id impl to the server Isn't there a famous saying about this? Something like: > Every problem in XMPP can be solved by adding another ID, except the problem of too many IDs ↺

We don't want a new ID though. Just to provide more structure with the existing one