XSF Discussion - 2024-08-26

sort by time ?

clock don't run backward ;p

It do in distributed systems 💀

oh shit

moparisthebest, or, does anyone remember the first AMD dual-core processors where the time counters weren't synced between the cores? caused all kinds of fun bugs

> clock don't run backward ;p Someone told that to NTP yet?

Time to write XEP xxxx: NTP over Pubsub

kurisu, you simply store message the way they arrive, if you request only in one direction

if you request in both direction, backwards and forwards, you load message sorted by time, and pray that the server has subsecond timestamps (which basically means no message will ever have the same timestamp)

if the server does not have subsecond timestamps, you can decide to still life with messages loaded in the wrong order which have the same timestamp

or you implement a very complicated logic how you store the order info, which to my knowledge no xmpp client has implemented till now

whatif server had a power cut and pushing rubbish ? will Donald Trump save Elon Musk for Harris ?

> kurisu, you simply store message the way they arrive, if you request only in one direction Well, you also have messages that are just sent to you directly, not out of mam. It may so happen that a live message is sent to you before the client gets older ones from mam. Is that handled at all? ↺

if the server has mam, i think it's *always* out of mam.

s/mam/MAM/

g

Uhm?

Nah. If the client is connected, it will not query mam generally. Only at the connect

A smart client won't use messages until the initial mam load is complete

To not mix it up

Or sorts and deduplicates by timestamp locally again

kurisu, you dont request message from mam you received live

client should sort by time is what am sayin' they get something that are "out of mam" they add to their local log. Should not concern mam at that point if i understand correctly.

a message has a mam-id, even a life message because its stored on the server first

you request messages *after* that mam-id

ok if you sort by timestamp its probably wise to update timestamps of live message with timestamps from the server

yeah tricky ..

generally there is no nice solution

MAM was never really good designed for that, the idea of MAM was that you only request in one direction

so never backwards, some stuff was added later to make backwards easier, but order is then always tricky if you cache locally

thats why Gajim only requests messages in one direction

My goodness

It really depends on your datastore. Getting it right in a flat file for example, is trivial. In a sql database that has no natural ordering it is trickier to do the synthetic sort ids needed and so timestamp is "close enough" for most people

I've even seen clients requesting mam based on timestamp instead of based on id, but that's maybe a bridge too far in general

If we're limiting ourselves to one way, it's no harder with sqlite than it is with a flat file, just use the default rowid which autoincrements and thus preserves the order.

But if we're doing both ways... With sqlite one could find the lowest counter field value and decrement that I guess... While with a flat file prepending would mean overwriting the whole thing from what I understand

prepending is no issue, though yes it may have performance cost. the problem with decrementing is that negative ids may or may not be allowed. and then there's the mess if you allow "holes" which is where it actually gets hard

if you have only up to A and then you request backwards from Z to D ... you don't yet have B or C but how do you know that? and how do make sure to put them in the right place later?

That's why the server ids should be sortable.

That would magically solve every version of this problem, yes

What's this? We need another id on messages? 😁💀

the existing mam id could just be made meaningful

without a breaking change, creating MAM 2 ?

As a client how could you tell, because ^

it would just need to be a new feature var or something that means "the ids from this mam are ordered"

for the client to know

we can make the ordered without the client knowing first, no harm and some benefit, but yes we would need discovery eventually

Like even without discovery it would make debugging a lot easier for the human who knows

That seems like it'd be ok, next question is ordered how, easy if you mandate numeric, hard with strings

In the dawn of MAM, I argued for leaving IDs as opaque strings so servers could use whatever their databases uses for unique ids.

moparisthebest: UUIDv7 almost for sure is the right thing to do, but that can be up to the server

And the server is expected to have messages sorted by what?

I see the archive as a simple FIFO. Messages are added to one end, get removed from the other.

what's the point of uuivds oer just unit time with milliseconds ✏

moparisthebest: the server already has messages sorted

what's the point of uuivds oer just unit time with milliseconds or something like that ✏

what's the point of uuivds oer just unit time with milliseconds or something like that, which you can also sort by like a normal integer... ✏

kurisu: if you just want time you already have that

using uuidv7 gives you a stable total ordering even if two arrive the same millisecond

or using an incrementing integer server side gives you the same

or appending into a file!

> I see the archive as a simple FIFO. Messages are added to one end, get removed from the other. that's how the client always ends up getting them unfortunately. IIRC Jami pulls off a whole git-like tree thing to reconcile history, but that's because it's p2p. It should be much easier with centralized storage like jabber, but yet isn't entirely easy without a sortable ids ↺

Zash: yes, I meant in terms of ids, but yes

You can still infer ordering from MAM, you know you get messages in the same order (or reverse if you <flip-page>) and you know that messages from adjacent pages are adjacent to that

Tricky to translate into SQL, sure. Easier if you had a linked list I imagine.

yes, if you just have a linked list it's trivial

its also rather trivial to ad a sortable id for the server to the store and adding a feature to disco info ✏

What do people think about defining a new XMPP URI querytype for linking to messages? Something like xmpp:someone@example.com?ref;id=blah where id is either the @id or the stanza-id in a MUC case

to use in what situation? because we have replies

links to messages in another room once we have "spaces"

SavagePeanut: we already have spaces, but I don't think that's relevant here :)

yes, links to messages in another room is one of the ways. but also even links to previous messages, not as a reply just as a "remember you said LINK" etc

then you can type "Remember" and use a reply

is this purley about using reply without a body?

It's about linking to a message in a context that is not a reply at all

sure, if it's in the same conversation one could sort of abuse replies for this

the question is why a uri

a uri makes for me only sense if its used outside of xmpp

you want to click it in a browser

Why not both?

I suppose most of the inside-XMPP use cases I have in mind would be handled fine by message forwarding as well, once I support that

but yes link from totally outside can be useful

But even if you wanted it to only work in XMPP clients, wouldn't it still be a URI ? Otherwise how does gajim and Conversations link to the same post ?

not sure what i should do with a link, it somehow depends on the feature that any user can query any message from any MUC archive?

can i do that without joining?

if not, i dont think i will query that message

I suppose you'd join and query mam no ?

ugh 🤢️

I don't think it'd do anything without clicking and joining

why not simply forward the message you want to show someone

its all already there defined, you can simply start implementing it

A message alone isn't always helpful

yeah? and you can still do this with message forwarding

"We discussed this in another room, start reading here: <uri>" ✏

Yes, "this is where it was discussed: link"

a forwarded message, contains the whole message

a link is useless

> a forwarded message, contains the whole message But doesn't have everything after or immediatly before ↺

What message would you forward? The entire conversation ?

a link i can not preview, i can show an empty box and say "click here if you want to see what this is"

moparisthebest, the message i want to share?

Yes, so how all links work?

?! did you see the proposed link?

or are you talking about some link you have in mind but didnt tell us

lovetox: one message isn't helpful alone, often you need to link an entire conversation with perhaps hundreds of messages

not sure what you are talking about

the proposed link has one ref to one mam id

thats what iam talking about, but maybe tell us the feature you have in mind

Which the UI needs to scroll to with all messages before and after filled in

a forwarded message has also the mam id

.....

there is nothing you can put in a link, that you cant put into a forwarded message

There are important differences

are you seriously arguing that the uri spec is more versatile than whole xml spec?

the point is, i forward a xml payload (message) that has everything another client needs to display something useful to the user

a link has nothing i can display to the user

except an empty box that reads "hey someone shared something, please click here, and i will join a chat with 1000 people" and download 2 mb of data for you to get one message"

1. I can't put a forwarded message on my blog or project readme (biggest problem) 2. Forwarding you the message might violate security if you aren't allowed to see it. Sending a link would result in a nice "sorry you aren't allowed to see this"

2. At that point, copy/pasting it would violate security which is what people would do without forwarding it

People can always do that

lovetox: I think you've convinced me to implement forwarding first before I come back to deciding if I need this URI

These are real use cases that other systems have by the way (Sadly) many projects link to chat messages for documentation/downloads etc, slack, discord etc In slack itself if you send a link to a message and the user(s) don't have permission it'll ask if you want to forward instead

> (Sadly) many projects link to chat messages for documentation/downloads etc, slack, discord etc How do we do this with XMPP without a URI ?

prompote the use of forums!

SavagePeanut: did you mis-spell discord?

😭 https://support.discord.com/hc/en-us/articles/6208479917079-Forum-Channels-FAQ

Telegram has message links and those are occasionally used to link some posts from the web or even inside telegram itself. But then telegram also has chat history much more advanced than xmpp, it can do "breaks" in cache that singpolyma mentioned earlier today, so e.g. if you're following a link to a message sent a year ago, it will load only the messages around that message, and will load more as you scroll up or down.

oh and forwarded messages are also links at the same time there

kurisu: xmpp can do that as well (only load some around the message in question)

By XMPP can I mean, MAM can

Yeah in theory... But we discussed earlier today how awkward it gets in practice when you lack sortable message IDs

And in practice I don't think any client does that? ✏

But no client does that isn't an argument. If you now implement the telegram thing in xmpp no client will do that too. Seems even trivial https://xmpp.org/extensions/xep-0313.html#filter-time

Isn't an argument for what

> And in practice I don't think any client does that?

>seems Yes, *seems* as we discussed above.

For needing something else

It is one query to receive a specific part of history

With mam.

If you're not caching it, that can be simple I guess.

yeah. the "hard part" is disjointed storage

downloading and rendering is 'easy'

kurisu: the feature you're describing is mostly unneeded because most servers have a short rentention time for history (a few weeks / a couple of months)

It would be more valuable to e.g. find a way to sync mam between clients than your feature request

kurisu: unneeded i.e. not needed

I don't see how that makes it unneeded... I think you meant impossible ✏

> It would be more valuable to e.g. find a way to sync mam between clients than your feature request At that point you'd be reinventing whatever jami has been doing I guess ↺

> At that point you'd be reinventing whatever jami has been doing I guess It wouldn't be something you'd do everyday. It would be something you'd do when you get a new device and want to transfer your years of history

It is already possible to do this between two instances of the same client, but not between two different clients

I think nowadays people expect servers to store that tbh, especially given that years of history are usually a dozen megabytes

Without media

Yes. I think for community chat stuff that's pretty important these days

> kurisu: the feature you're describing is mostly unneeded because most servers have a short rentention time for history (a few weeks / a couple of months) I don't know that this is true, but even if it is, that's just a deployment decision ↺

What I love most about these "decisions" is that you never know

Which basically makes them random from user pov

Well user hopefully isn't making their stuff in a random server they don't know the policies of 🙂

Nobody reads "policies" of a server especially given that they're well concealed

If you don't know what a server is or what it can do why would you put your stuff there?

Yeah yeah in the end the users don't care. In the end they end up sticking to things like telegram because they know if you put something there it's there for good and won't randomly get erased whenever the admin saw fit

Whenever the admin saw fit to free the 10mb worth of your year long correspondence