XSF Discussion - 2021-03-31

Has anyone here though about or come up with a way to identify messages from bots? Seems like one can just include a special tag and namespace for it. Is there anything like that in the wild already?

jcbrand, bots should publish a bot identity in disco#info and clients should use that

jcbrand, https://xmpp.org/registrar/disco-categories.html#client

so you should send a disco query for every single message in a MUC?

not really practical in certain use-cases

'115 caches

or if that’s not feasible because MUC, you can just query when the participant enters the MUC

and cache the information until they leave it

You can really see that XMPP was designed without ever thinking about web clients

attaching such metadata to each and every message is incredible overhead

Hats

oh, I like that thanks Zash

We already have hats

<hat>🤖️</hat>

problem solved

😀

... *sigh*

lol

web making everything terrible again.

enjoy your ivory tower

jcbrand, to be clear, I’m not blaming *you*

I’m blaming the web.

jcbrand, out of curiosity: why doesn't the approach suggested by jonas’ work for web clients?

jcbrand, isn't this MUC being web-unfriendly rather than XMPP itself?

jonas’ yeah sorry, I don't want to be hostile, but the anti-web attitude in the XMPP community does grate a little sometimes. No big deal tho

jcbrand, fair. Let me state for the record that I am really glad for your work on converse.js, because despite the web being terrible, XMPP-IM needs a web client which is *not* terrible and you seem to be doing a fairly great job at that.

what jonas’ said, 👍 for jcbrand

aw shucks, thanks guys

It's really the anti-everything-except-the-web attitude of the web that irks me.

Sure, I get that

web-pacman.gif ✏

flow: my knee-jerk response is that in various situations you can't cache/store everything across sessions like you can with desktop clients

jonas’, now I need you to drop everything and make that gif. you may edit the systemd-om-nom-nom.gif 😉

jcbrand, but hypothetically, if you did the disco#info on the first message from a participant and cached that for a session (or until that participant leaves), that would be feasible and good enough, wouldn’t it?

jonas’: yes, but that is then not XEP-0115 compatible right?

I’m not sure '115 even works in MUCs

In theory you could cache caps-hash → disco response forever

if it did, you could sometimes skip disco#info lookups if your session has already seen the hash though if '115 worked

Zash: Yes, but in web clients you can't ✏

even that can do a lot (e.g. if the bot only hops into the room briefly and you don’t know the real JID)

You say that including a tag is wasteful, but in a MUC with an occasional bot post and lots of posts by other users, it seems wasteful to me to do a disco-info for each new poster

it could be a tag on the presence, not the messages themselves

But it's ok, the hats thing will work for my current use-case and for the open Jabber network doing a disco query for each new occupant is probably also fine

not each new occupant but each new message author

jcbrand, you probably can re-use the disco#info information for other purposes too, so I am not sure if it is that wasteful

Unless you want to identify bots in your occupants list...

seems very chatty to me. On a desktop client you can cache all this fairly aggressively, but on web you need to do potentially hundreds or thousands of disco info queries for every session

On web there is IndexedDB which doesn't have a size limit, so in theory you can use that and then your webapp is more like a desktop app, but having integrated that into Converse (via localForage), I've seen that writes are suuuuper slow.

And you can't use IndexedDB in all cases

I have some ideas on how to improve performance on Converse (basically batching writes), but haven't had the time to iron out the kinks

jcbrand: what about localStorage?

localStorage has a 5MB limit, so if you have lots of MUCs it fills up pretty quickly

Now you could regularly delete older messages, but then you lose your OMEMO history

So far that has made me reluctant to aggressively get rid of older messages

I can't actually use Converse.js anymore, it runs into those limits in a few minutes 😕

If it wasn't for OMEMO then it would be much easier

Not while joining all MUCs anyway

Zash: If you're up for it, you can try the master branch with IndexedDB

performance has improved a lot recently

Still quite a bit slower than localStorage, but usable IMO (although not usable in some use-cases like having it integrated into an existing resource-hungry site)

I might even make IndexedDB the default in the 8.0.0 release (especially if I can get the batched writes to work well)

One advantage that disco has over hats is that hats are per MUC, so you have to assign them for every new MUC the bot enters

Hm, reminds me of the issue of MUC-component wide affiliations and such.

yeah, would be great to have

is any of this fixed or improved by MIX?

Not that I expect MIX to supplant MUC any time soon

> eric: for which > screen size is certainly still a problem for KDE apps, like with plasma mobile the wifi setup dialog, no one had tested it with WPA2 Enterprise, the OK/Cancel buttons were off-screen, I had to hook it up to a monitor to set it up moparisthebest:

I thought web apps could request more localStorage space nowadays?

eric: ah, no, but in general there are many more bugs than things that actually work in Linux phone land so I'm just waiting it out :)

Reminder that XMPP Office Hours for this week are today, not Friday! "Soprani.ca: bridging us all together" by Stephen Paul Weber (singpolyma) on Wednesday, 31st March 17:00 UTC

Office Hours for this week (about jmp.chat) is starting now! https://socialcoop.meet.coop/sam-pku-dud-niv

Ge0rG, why are groupchat messages in an user archive a bad idea?

flow: because the client needs to support that, be aware that those come from a MUC, and treat them differently. I think that the MUC MAM approach has worked better.

Most clients today will probably do weird things when receiving groupchat messages from MAM

Ge0rG, thanks for the reply. But I've to admit that this appears to be more like a client issue

There's also the problem with how a MUC will send one message for each joined resource, which complicates server processing.

Way simpler for the MUC to do it, as it already knows when processing that it's a single message, before it broadcasts it.

flow: a MUC will only send messages to you while at least one client is connected. If you only have one client, you will only receive from MAM what you already had before. If you have two clients, you will receive a random subset of the room history with no way to know where there are holes

There are scenarios where it's desirable that the user only sees history from when they're joined, but it does get weird and should probably be dealt with on the basis of affiliation.

It's getting even more weird with MIX MAM, and I've been telling that for years.

BTW, saying “no-one does this” isn’t correct :)

This is the point where I mention that experiment I did in rewriting outgoing MUC joins so that they come from the bare JID?

But you’re right that it doesn’t work terribly well.

Kev: I'm eager to read your implementation and interop story on the ML

Ge0rG, what about MIX MAM ?

Zash: you could convert that from a 20% implementation to an 80% implementation, and I'd actually test it

Zash: s2s isn't flawless, it will cause holes in MAM

What's those 60% then?

Zash: tracking outgoing presence, leaving rooms, nickname management, bookmark sync

If we have a MAM meta-search that searches each archive, we could probably get away without having mix history in personal archives.

Ge0rG, but how will I be motivated to work on it without at least someone testing it?

Zash: well, how am I supposed to test it if it breaks everything?

Someone needs to go push the stalemate resolution button!

And there's like 3 different bookmarks methods atm, that seems fun to deal with

Ge0rG, track what outgoing presence?

Zash: room joins from clients?

That part should work afaik

What about error handling?

That goes under "leave"

which is under TODO

I think it even dedups additional joins, tho I'm not sure that's optimal.

Zash: it's a server module in a state I can't possibly deploy on a production server, and I'm only using one single room from my other server

Should set up that bleeding edge Prosody instance for client devs ... one day.

Yeah, crash me!

What rules would be needed? Just rewrite anything going to/from a MUC so that the user part is bare?

Determining it’s a MUC by the first thing that’s sent being a presence with muc in it?

Kev: you need to implement a bouncer that will multiplex the room between all your clients

Oh yeah, that.

So, basically imprementing MIX-PAM, in other words.

Pretty much

Kev: kinda sorta

Well, it's probably not so much to implement, and doesn't need to persist anything. But it might have to do sophisticated things with room history generation on join

Keep track of joined rooms, joined resources to deal with rewriting in- and outgoing stanzas. Then subject and participants which lets you add new local resources and send them a synthesized join presence flood.

So, it’s much worse that just implementing MIX-PAM.

No, it's better because it works with MUC, which is what Everyone™ is using today.

With zero changes required of clients

I mean it’s much worse to implement.

And with a manageable overhead on the server

Sure, it's MUC

You know what it is like implementing, actually?

It’s basically implementing FMUC.

Which is a MUC implementation that can join another MUC.

FMUC-on-your-JID

Because we need more things-on-your-JID! 😀

It’s the bastard child of MIX-PAM and FMUC.

And implementing it will give a significant immediate benefits

Mobile clients don't need to self ping any more

Now the server will need to self-ping! 😀

You'd get reasonable push integration

Less presence flooding of rooms

Zash: the server knows about its s2s, so it only need to self ping after s2s went down

Could probably do the presence versioning thing, closer to the user.

Zash: needs new protocol

Hm?

Kev: that solution is 100% transparent to existing implementations both on the client and on the MUC

In theory you could do the same thing but have it speak MIX

In theory you can do anything with enough monkeys and typewriters, yes :)

No idea if MUC-MIX translation is easier on the users' server or the MIX host.

MIX. Much :)

Ge0rG, FWIW I think it's okay to leave the door open for storing groupchat messages in the users local archive, but Something Needs To Be Done first.

Zash: yes, a new XEP needs to be written.

No place in 313 for this

Is it worth having to bump the namespace for the sake of changing the SHOULD to the MUST?

Ugh

I remember having to explain to another developer not to do it like written in the XEP

Kev: which one?

MAM.

Let's not bump any namespaces

I think changing a SHOULD to a MUST would require that.

SHOULD NOT then?

or perhaps a PLZ DONT

Kev: there are 14 SHOULD and a dozen should.

So which one are you talking about?

The one closest to the word 'groupchat' I would have presumed?

So... > A server ~SHOULD~ MUST also include messages of type 'groupchat' that have a <body> I really hope that's not the intention

You proposed onlist changing into MUST NOT store gcs.

I think.

Kev: last time I checked, MUST NOT was the opposite of MUST, so please excuse my confusion. But yes, I suggested that.

As an escalation of "SHOULD NOT", but I like strong words.

I also looked up that the first time I wrote "MAM subscription" on list was in 2015, and probably even earlier in here.

This is a wordification of what future (currently unreleased) Prosody will do, if nothing changes before the next major release: - **Do not** store messages of type headline since these are supposed to be for transient notifications, most often PEP events. - **Do store** messages of type error, since if you sent a message it is of interest to know that delivery failed. - **Do not** store messages of type groupchat, as MUCs will send one message per joined resource and most often provides their own MAM. - Follow \[XEP-0334: Message Processing Hints\] advising for or against storage. - **Do store** messages with a `<body>` and/or `<subject>` element, as these carry messages for users. - **Do store** encrypted messages for the same reason as with `<body>`, as indicated by \[XEP-0380: Explicit Message Encryption\]. - **Do store** messages with \[XEP-0184: Message Delivery Receipts\] requests, and the receipts themselves, as if something is important enough to need such a receipt it is probably important enough to archive. - **Do store** messages with \[XEP-0333: Chat Markers\], for the same reasons as with receipts. - **Do store** MUC invites, both mediated and direct. - **Do store** messages with an \[XEP-0353: Jingle Message Initiation\] payload, as users will want to know if they had missed calls. - Anything not covered by this point is either something new that may or may not warrant adding to the archive or something that should not be archived.

And I don’t think you’re even the first person to suggest mamsub ;)

Ge0rG, I like you response on the ML, but you may want to consider adding reasons against groupchat messages in a user archive. I think your mail contains none, and the reasons are good. I feel we often discuss on the mailing list stating "everyone knows that X is a bad idea", without actually providing any reasons/arguments why it is so. And that probably deters people from participating.

flow: thanks! I'm not going to be at a proper computer for the next days, so feel free to summarize, link or copy paste what we discussed here today

If you were to do dedupe on the server, or whatever, you could also achieve reasonable groupchats in the archive behaviour.

And probably for less work than trying to implement the MUC-PAM discussed earlier.

That's a big and messy if tho

Unlike finding a way to do dudupe and everything else involved in MUC-PAM? :)

I’m just giving another reason that I think MUST NOT would be excessive.

Just Unfork Stuff™

Kev, I agree tho, if you do go and solve that and other issues, storing groupchats in the users archive is probably okay.

Kev: also please solve reliable s2s while you are at it. And World Peace. Whatever is easier.

You know what, let's flip that default

Kev: you need additional protocol to get MUC messages into the user account, even when no client is connected, which is surprisingly close to MUC-PAM

Without that, you get invisible holes in MUC history, which is the opposite of what xmpp needs

not sure if we MUST keywordify here, it appears a simple "we found that user archives with type=groupchat messages cause trouble if not taken special care of, and clients may want to consider ignoring type=groupchat messages in a user archive"

could be sufficient

Makes sense to me.

I still think that servers shouldn't send all the MUC garbage to a client

Looks like XMPP infringes on this patent in the USA, </stream:stream> illegal: > The claimed invention is a system where at least one robot “generates a goodbye command that terminates a communication session” with another robot and “relinquishes control.” https://www.eff.org/deeplinks/2021/03/stupid-patent-month-telehealth-robots-say-goodbye

Is it April Fool's already?

Unfortunately no