jdev - 2025-05-11

Something great (frustrating) is that I have no idea if I'm missing chat history in any MUC I'm in, while my computer isn't on. My phone is connected so it would just send it there, but now I don't have that history on my computer!

MUC history you get when joining and/or doing a sync. It's not tied to being joined?

Haven't the slightest idea, no way for me as an end-user to tell. And plus Gajim didn't show this room's history but it got future@joinjabber.org's easily, and Monocles on my phone got both easy, faster than Gajim did even, when I first joined both of them. Gotta love the XMPP inconsistency

You are always missing messages as this room exists since a decade, do you want to download a million messages?

I implement upgrading PMs to MUCs on the same (virtual) JID and vice versa. Tkabber, Dino and another.im survive it rather well, but Gajim fails: ``` ValueError: Trying to add GroupchatContact -566866516@zg.94.177.207.80, but contact already exists as -566866516@zg.94.177.207.80 (94.177.207.80) (in roster: False) ``` Even unsubscribing manually doesn't help, as this "roster: False" seems to mean that just a chat exists in some internal structures, only deleting it completely would help perhaps. Is this behaviour correct? I did already wonder if this was a good idea at all… no big problem to make a different JID space for MUCs, I just wanted to make the migration as seamless as possible.

You probably need to delete the cache ✏

.cache/gajim

And yeah gajim does not react well if you add groupchats to the roster ✏

Very bad idea btw

A jid should always point to the same thing

Im not sure what you mean by upgrading a jid

> Haven't the slightest idea, no way for me as an end-user to tell. And plus Gajim didn't show this room's history but it got future@joinjabber.org's easily, and Monocles on my phone got both easy, faster than Gajim did even, when I first joined both of them. Gotta love the XMPP inconsistency It's an extensible protocol, not a consistent or reliable one. For those would be CMPP and RMPP ↺

>> Haven't the slightest idea, no way for me as an end-user to tell. And plus Gajim didn't show this room's history but it got future@joinjabber.org's easily, and Monocles on my phone got both easy, faster than Gajim did even, when I first joined both of them. Gotta love the XMPP inconsistency > It's an extensible protocol, not a consistent or reliable one. For those would be CMPP and RMPP 😜 ↺

doge: this has nothing to do with reliability, user is just confused that different clients provide different capabilities of querying old messages

You could say that about any sloppy software. "Windows BSODing has nothing to do with reliability, user is just confused that different operating systems provide different capabilities of running continuously"

No not really. Only because a client does not meet the expectations of a user does not make a protocol unreliable

It maybe makes the client unreliable

The extensibility can make it as relaiable as one wants after all 😇

halscode was talking about inconsistency. Consistency I think is one of those things that extensibility cannot achieve and moreover tends to harm.

But then reliability also depends on consistency to a degree I think. "Can I rely on my contact's phone to show them effectively the same thing I see?" - even if all xmpp clients perfectly did their job but in their own way, supporting different extensions, the answer would still be no, because you don't know what your contact's phone supports. E.g. you wouldn't know if he sees your pfps, because he may have a client that doesn't support it. And as of right now, people including me randomly don't see other people's pfps even on clients that support pfps

What is pfps?

Avatars

OK. Thanks. How do you turn avatars into that acronym?

i guess we should ideally be able to trust that messaging clients agree on and implement everything documented here: https://docs.modernxmpp.org/client/protocol/

so it's not an entirely arbitrary set of features

Martin, "ProFile PictureS"

Thanks.

> Haven't the slightest idea, no way for me as an end-user to tell. And plus Gajim didn't show this room's history but it got future@joinjabber.org's easily, and Monocles on my phone got both easy, faster than Gajim did even, when I first joined both of them. Gotta love the XMPP inconsistency So you're saying you joined this room with gajim and got zero recent messages at all on first join? ↺

Gajim requests only 24 hours on first join

In slow channels this could be zero

We're we really silent in here for 24 hours? I miss the old room history API where the room would just send you whatever made sense on first join...

Yes, this ^ and Dino not requesting something from before the fist join often gets confusion by users not used to this

This is could be better on protocol level but there is no way to request an amount of messages with user content

If I request 100 messages, it could still mean zero as chat states are also messages

Hm. But the client could count how many have a body and stop requesting after it got enough with one?

Yes, but this is of course more complex to implement

“03:02:05 1 days, 10 hours passed…” yes, we were silent for over 24h.

> So you're saying you joined this room with gajim and got zero recent messages at all on first join? that's correct. It did try to, it spun on something along the lines of "getting recent messages" for maybe 30 seconds before giving up. ↺

> Gajim requests only 24 hours on first join tbh a better implementation would be by number of messages, i.e. 100. Could do "or past day, whichever is more". Not sure if MAM-MUC supports that though ↺

I think for Spades, when I get to MUCs, I want to try to do infinite backscrolling, where when you scroll to the top it loads more messages (20 or 50 at a time, possibly depending on if the connection is metered). And it would also do this on first join of an MUC of course, possibly with even more (i.e. 100). This would make good use of XEP-0059

Note: there is usually a limit to how much MUC history is available, and you'll want to dip into your locally saved history where available ✏

> halscode was talking about inconsistency. Consistency I think is one of those things that extensibility cannot achieve and moreover tends to harm. Most of the inconsistency in XMPP is due to competing extension standards (profiles, avatars, file attachments...), and some of it is due to widely differing implementations. A common SDK like DeltaChat has can actually mitigate most of these ↺

Hey speaking of which, where do I find the Snikket SDK? I'd forgotten it existed, but it could be useful now as I'm thinking of (experimentally) developing an XMPP-based (non-chat) app

> I think for Spades, when I get to MUCs, I want to try to do infinite backscrolling, where when you scroll to the top it loads more messages (20 or 50 at a time, possibly depending on if the connection is metered). And it would also do this on first join of an MUC of course, possibly with even more (i.e. 100). This would make good use of XEP-0059 Yes my SDK does this so even if initial sync of 3 days got nothing a would get last page when you look at it. But this discussion has me thinking I should switch from 3 days to 1000 items on initial join ↺

badrihippo: things are in flux but we're in process of moving it to https://borogove.dev

Obviously it's simpler to grab a huge tranche, but progressively fetching small portions until you have 'enough' would be preferable and allow for better responsiveness

Then you're also not fetching days of messages you already have stored locally anyway

If you have anything stored locally anyway this isn't relevant

We're talking about first join

As soon as you have local history you can fetch since then

Or before, if the user scrolls up

Right

Isn't it still possible to have "holes" in the middle? Maybe you have message history saved locally from November through January, but you haven't logged in since then and now it's May. So when you log in in May, maybe you don't want to get the whole message history in between, that could be a lot of messages. So as you scroll back, you'll see your new messages from May or April, then you might need to look up messages between May or April and January via MAM, and then you have some history again maybe, and then you need to fetch more from before November... it's pretty complex but it's worth figuring out

Yes, holes are a problem, so it's best if your datastore can represent them, and then you can fill them as needed

Yes if you choose to have holes that's a whole different kind of problem and set of solutions

> Isn't it still possible to have "holes" in the middle? Maybe you have message history saved locally from November through January, but you haven't logged in since then and now it's May. So when you log in in May, maybe you don't want to get the whole message history in between, that could be a lot of messages. So as you scroll back, you'll see your new messages from May or April, then you might need to look up messages between May or April and January via MAM, and then you have some history again maybe, and then you need to fetch more from before November... it's pretty complex but it's worth figuring out Yeah that's what I stumbled upon when trying to implement my own client. ↺

> 2025-05-11T07:13:36Z - lovetox: > Im not sure what you mean by upgrading a jid Groupchats were simulated via PM previously, now I rework them to work via MUC. ↺

>> 2025-05-11T07:13:36Z - lovetox: >> Im not sure what you mean by upgrading a jid > Groupchats were simulated via PM previously, now I rework them to work via MUC. The what ↺

Oh you mean like you have your own corporate system in place or smth?..

> 2025-05-11T16:33:09Z - doge: > The what The transport. I don't know why was it decided to be done so, TBH. But it definitely has its benefits. F.e., messages are delivered for sure if there is a subscription, while joining all the existing MUCs via bookmarks might be unreliable depending on a client. Also, no memberlist and no excessive traffic for it. I'm going to keep this legacy system too during the next major release at least. ↺

What transport is that

Vk or mailru? Love that ancient avatar

> 2025-05-11T16:49:27Z - doge: > What transport is that telegabber ↺

> 2025-05-11T16:49:44Z - doge: > Vk or mailru? Love that ancient avatar The meaning of the avatar is quite opposite lol. ↺

but what did you do with PMs ? i dont understand what it means to simulate groupchat with PM

> 2025-05-11T18:04:28Z - lovetox: > but what did you do with PMs ? i dont understand what it means to simulate groupchat with PM Just sending messages from all the members with fake text headers. Like this: ``` 06/01/24 | 19:58:06 TDLib chat: ⬅ 139874795520 | Hi， 未来 (@anl2018) | reply: 139872698368 | - Sobhan (@SobhanCU) | 不。 what？ 06/01/24 | 19:58:31 TDLib chat: ⬅ 139875844096 | - Sobhan (@SobhanCU) | reply: 139874795520 | Hi， 未来 (@anl2018) | what？ 'no.' 06/01/24 | 20:00:02 TDLib chat: ⬅ 139876892672 | Hi， 未来 (@anl2018) 兄弟，你来搞笑的吧！ ``` ↺

MUCs will provide a more native integration, especially for OOB messages where there is no room to put some text (that's a huge downside of OOB actually).

is the difference between a PM and normal groupchat message not just the "type" attribute? both come from a full jid

im glad more people are making clients lately

maybe we'll get some good ones

i dont think gajim should have a problem with it, of course the message would need the <x> element, otherwise gajim would not discover that this is a PM

then the bare jid gets treated as an account, if later conflicting information arises that would somehow make the same bare jid suddenly a MUC stuff breaks

> 2025-05-11T18:24:18Z - lovetox: > is the difference between a PM and normal groupchat message not just the "type" attribute? both come from a full jid There's indeed no huge difference. Just the type, and the resource (the outgoing resource was always the same before). Though AFAIR, the MUC spec mandates that when a messages comes from a resource, a presence about the following member should come first, otherwise the message should be rejected. ↺

but this is not necessary for a PM

> 2025-05-11T18:30:12Z - lovetox: > then the bare jid gets treated as an account, if later conflicting information arises that would somehow make the same bare jid suddenly a MUC stuff breaks Yup, currently `_get_contact` returns the non-groupchat contact despite the `groupchat=True` argument, and the code breaks further because it attempts to access groupchat-specific properties. Yet again, I didn't upgrade it to the latest version yet. ↺

> 2025-05-11T18:33:34Z - lovetox: > but this is not necessary for a PM Are you talking about possible MUC-to-PM downgrades? ↺

a PM is a message with type=chat and <x xmlns='http://jabber.org/protocol/muc#user' />

everything else is *not* a pm

> 2025-05-11T18:34:17Z - lovetox: > a PM is a message with type=chat and <x xmlns='http://jabber.org/protocol/muc#user' /> Uh, you mean MUC PM then. ↺

what did you mean?

What is plain 1-to-1 messaging then?

a chat messge

Okay then.

private message has special meaning in XMPP and always means MUC PM

I just think "chat" might be also confusing because it might mean group chats as well.

no, usually you name the message after the type attribute

type=chat -> chat message type= groupchat -> groupchat message

pm is a special case of a type=chat message

anyway, im more confused now

Nevermind anyway, I should make MUCs on a separate JID as a safety measure. Even though such an upgrade works well in some clients, there might be unpredictable consequences in arbitrary legacy ones. Not needing a separate domain is lucky enough already.

It would just complicate working with JIDs a bit more (rn, it's assumed everywhere they can be easily converted to numbers and back everywhere). ✏

but anyway, i made a bad design decision once when assuming groupchats are not in the peoples roster

xmpp terminology != traditional terminology

since then it always breaks when some client for whatever reason adds a groupchat to the roster

halscode, i agree with you in most cases, but know your audiance, this is a xmpp developer chat

yeah ik, just pointing out that it's not the same. so it's gonna be confusing

> type=chat -> chat message > type= groupchat -> groupchat message as far as xmpp goes, for instance, this is right and a good way to differentiate, because the subtle difference is important ↺

it works okay for end-users too as long as they're delineated in separate chats. but PMs get a little confusing (and on a side note, this has inspired me to call them "whispers" in my client to convey that, generally nobody else is going to see it but a sneaky eavesdropper still might, as opposed to a "private message" which is misleading or a "personal message" or even "direct message" which is confusing and doesn't differentiate enough from regular 1-to-1 chats. Especially as a "whisper" might be done in a public place to not share something with everyone there) ✏

Oh something else I just thought of: I'm planning on implementing pinned messages into my client, at some point. It's something my family group chat uses a lot to pin important information like schedules. Is there already an XEP for it or will I need to write one?

I think those annoying messages I used to see in older gajim versions "xxx set group subject to yyy" are roughly that

halscode, there is no spec to pin messages yet, but you can use the groupchat subject

also you need server support for it to work

thats nothing that you can implement just clientside

probably would work more for like "mark as urgent" or something like that

It's similar but different. Pinned messages may be somewhat less important but something you want to reference easily. Occasionally it's also useful in direct messages

halscode what is your client you're working on ? :) Curious about it

so clients know to treat the message as something to be "pinned"

> probably would work more for like "mark as urgent" or something like that see that sounds like something that would change the appearance of the message to emphasize it ↺

can I instruct MAM to not expire certain messages?

halscode: what's your client?

> can I instruct MAM to not expire certain messages? nah doesn't seem to support it ↺

> can I instruct MAM to not expire certain messages? iirc there isn't even currently a way to tell how long mam stores your messages.... ↺

> halscode: what's your client? I'm using Gajim and Monocles, and I'm making one currently called Spades: https://codeberg.org/spades/spades-flutter (the readme is ancient, I've got to fix that some time this week when I have the time for it) ↺

it might be accomplishable with PEP, maybe?

there are many ways to the goal, but all include server side support

looking at PEP, it's probably doable that way but only for direct messages, not groupchats

in single chat, you can just make that feature local

though i would not make it as "pinned"

rather i would implement a "mark this message, so i can look it up later" or something

and then have somewhere a screen, that shows you all marked messages

flag as urgent

something generic like that

> rather i would implement a "mark this message, so i can look it up later" or something Kinda like a "starred" message in the Google Messages app ↺

Okay for one on one, though for groups it's more difficult

in groupchats you pin the message for everyone

that needs a mechanism to distribute the information to all members, hence you need server support

the subject distribution is that mechanism, we just need to have a spec that we can extend the subject with the message ids

> private message has special meaning in XMPP and always means MUC PM I'm not sure that's true 😛 but it definitely sometimes means that ↺

> that needs a mechanism to distribute the information to all members, hence you need server support it works for like if everyone happens to be on at the same time ↺

or always syncs the room history immediately

`<my-cool-pin xmlns="tag:my.example,2025-05-11:pin" id="some-message-id"/>` ✏

Can a pinned message just be a normal <message/> with some flag saying it's to be displayed as pinned?

it'd probably look something like that^

For example, on Telegram you can even see a history of pinned messages that is you can jump to the previous pinned message.

no thats not how you would do this, the message is not modified

you just attach the message ids you want to pin to the subject for example

if you make it a flag in the message itself this would mean all clients need to download all messages ever to find all pins

further your approach would mean a user needs to know that a messages needs to be pinned, before writing it

also it would mean other persons cannot pin messages, only the person that writes the message initially

no yeah that's what the `id="some-message-id"` attribute would do

a separate message saying "The referenced message is important and should be shown in a 'pinned' or similar interface" ✏

From my observation, messages are the most reliable part of XMPP. They are delivered most of the time while even profile pictures randomly disappear for people, nicknames won't change etc.

maybe even have like a `prev="some-other-id"` to reference the *previous* pinned message (as known by the sender)

> maybe even have like a `prev="some-other-id"` to reference the *previous* pinned message (as known by the sender) that wouldn't work actually thinking about it unless the message was like pinned recently after pinning ↺

wouldn't work as like a "click here repeatedly to step through the pins" kinda thing

> that wouldn't work actually thinking about it unless the message was like pinned recently after pinning pinned recently after sending* ↺

In practise you almost certainly don't want to pin by ID, but rather make a copy. Otherwise if that message expires from the archive or similar... Sure you could specially protect it but then you need really specific support from the server which is quite gross. Our one pinned message is a copy now, it's just vaguely text only and only one which are the limitations we may need to overcome

singpolyma, dont agree. This would make it *much* more complex, basically you are asking all clients that support this, to be able to restore a true xml representation of messages they have in the database

whatfor? so the server does spare setting a flag on a database row that prevents it from deleting?

Doesn't need to be "true" just needs to look right when you render it 🙂 in fact for an omemo MUC you *need* it to *not* be true... And also can't use OMEMO at all for the pin which is a bit of an issue

i dont see here anything "gross" server side

Would mean the server would need to know about pinning. Vs just being a bag of data (pubsub, MUC, etc)

he gets a message, and parses some ids defined in a XEP .. the most trivial stuff imagineable, at the same time you dont have any doubts asking many devs to restore message xml representations for this feature

then you open up the problem that the client that pins dont even understands everything in the message he tries to pin, meaning he cannot restore everything

for example a file attachment

Sure, but that client wouldn't understand that anyway even if it were present

then you have the abuse vector, where you can basically create every message in the name of someone else and pin it

> Sure, but that client wouldn't understand that anyway even if it were present ? ↺

i pin a id, i dont need to understand any content at all

Assuming you have permission to pin. But yes I suppose that could be an issue in some cases

if for some reason server cannot store the messages longterm, then i still want only send an id, and the server should simply attach the whole message as copy

as servers store the xml anyway

but i really doubt its a big problem for the server to preserve some messages

When asking whether to implement something on the client or server, part of the trade-off is when do you want it

If you are ok not having it for 10 years when the last Debian stable server upgrades, then server is fine

Otherwise you want to do it on the client

Client is always better except for security which obviously must be done on server usually

Tend to agree

Partly because in 10 years everyone will want something else

dont see the relevance here, this cannot be implemented without server

if you store ids or message copies, you need to modify the server to store your new data

Do it a way that you don't, pubsub ?

No server is putting pubsub on MUC jid right now, so you'd need to pick some pubsub. Which is annoying but doable in a pinch

and then your problems start, you dont have a permission model that maps to MUC permissions, again even more server work