XSF Discussion - 2018-07-27

Andrew Nenakhov: if your company only employs jabber fans, this is an expected outcome. If you start with a bunch of people who never before did business chat, slack is much easier than any of the alternatives, especially xmpp

Andrew Nenakhov, to be frank, I don’t like how you seem to do decent development, but don’t discuss with the community whenever you make such choices.

(re 0221 vs. 0066, but also MUC etc.)

(although muc is kinda understandable given the state MIX is in)

> Zulip is 100% open source software, built by a vibrant community of hundreds of developers from all around the world. With 120,000 words of developer documentation, a high quality code base, and a welcoming community, it’s easy to extend or tweak Zulip. What did they do right that we did wrong?

They built a service, we build a protocol.

We need a service to champion the protocol.

Yes, but where are we going to get the "vibrant community of hundreds of developers"?

Same thing, I think: By building a service. Their community is building on one client and one server. We have a fragmented ecosystem where lots of devs work on lots of clients and servers.

That's fine, but we need some bigger ones with commercial support too.

Ge0rG: It's fairly rare for the hundreds of developers to be doing much, it's probably a handful of core contributors

yah, that's probably true as well

waqas: so where is the modern xmpp client with a handful of core contributors, then?

The vibrancy though, we need to learn to vibrate appropriately

We could start out by making more of a buzz

My phone can vibrate, if that counts.

Hi. Is there a way with OMEMO to specify xml:lang?

re vibration: https://www.youtube.com/watch?v=JWAAbmps_Zs

jonasw, to me it seems that coding > discussing. No point to argue how to do something until we have a working prototype. If there are competing prototypes, community might stick to the one that is better, the other ones will die out by themselves. As is the case with MIX, creating a protocol without implementation might be a perpetual affair.

I think there is a balance to be struck between the two

"Rough conensus and running code" is a decent mantra in that regard

Andrew Nenakhov, fair enough. this wasn’t necessarily meant as an accusation by the way, although I can see that it may read that way. I think this might also be a symptom of the issue that we have way too many ways to do the same thing.

e.g. I looked at MUC Sub recently, and it definitely suffers from code-first problems, a number of things are not specified adequately, or at all

> Same thing, I think: By building a service. Their community is building on one client and one server. We have a fragmented ecosystem where lots of devs work on lots of clients and servers. Currently what we're trying to do is to build server AND clients for all major platforms, so they simultaneously support new features and do it so that all clients have a similar look and feel

Andrew Nenakhov, for example, any reason why you’re not using XEP-0385, which seems like a more natural choice for transmitting that type of stuff?

Because it has a way too big number and wit myriad of xeps it's hard to read up to that point.

And it looks kinda vague

:)

uh, I haven’t read all XEPs up to that. I follow more of a mix&match approach ;-)

what is appealing about 0385 to me is that it has a well-defined way to share multiple sources for the data (like XEP-0221) has, it is reasonably simple to implement, and it allows to provide metadata like file size, mime type and dimensions

We slightly extended 0221 for exactly that.

meh

why not use something which already exists instead of modifying something else which ~nobody else uses?

https://xmpp.redsolution.com/upload/4bddf4f264f5c6577f16551f16a0abdf3f7ff84d/sHmWWFaYLaa0hl29XM0iYHhDMe07ouSq3B0jrZF6/IMG_20140212_165807.jpg https://xmpp.redsolution.com/upload/4bddf4f264f5c6577f16551f16a0abdf3f7ff84d/AccuS9gfadNqhr46nO3c6JokNJQ85sM3ZMmoSSfn/Screenshot_20180727-031120.png https://xmpp.redsolution.com/upload/4bddf4f264f5c6577f16551f16a0abdf3f7ff84d/Hmg5tqjf5dxZOYWngVKfMEmnhgkYOP55kkb96iqh/IMG_20140212_165719.jpg https://xmpp.redsolution.com/upload/4bddf4f264f5c6577f16551f16a0abdf3f7ff84d/V1e5V9S2fBwTzTEGPi08gkLMbcuLDsQC4jypTxc0/4b92c4f3-2cd6-4a75-ae43-cb1abc98ddf2.jpg

Here are 4 files. All with sizes, dimensions, file types, etc

jonasw, because no one generally uses xmpp at all. So being compatible with some obsolete clients that are a long time abandoned is futile.

Andrew Nenakhov, there are actively developed clients outside of redsolution.

We have a clear picture of the paying audience we plan to reach. They'll be happy to use our clients exclusively. If we succeed, our approach will be standard. If we fail, well, other clients approach will win.

Andrew Nenakhov: your approach will not be "standard", it will be a silo.

I expect it more to end up like whatsapp with no compatibility to other clients at all, yet another XMPP fork.

or slack.

or any other cloud-based chat offering with a half life of five years.

Ge0rG, > Andrew Nenakhov: your approach will not be "standard", it will be a silo. X in XMPP stands for extensible. It seems to me that we understand Extensibility differently. If our approach succeeds with users, we'll suggest updating the standard. And we do everything with providing compatibility, not breaking anything anyway.

Andrew Nenakhov: what MattJ said. "rough consensus and working code". Without consensus from the community, your code will end up in a dead end

Andrew Nenakhov: I can understand your position from a business point of view, but I'm not yet sure it will actually benefit the XMPP community

Andrew Nenakhov, while you figure out your business model and acquire customers, the community is doing its own development. then we end up with two strains of XMPP, one of which in your company which isn’t written down in standards, and one used by the community actually written down in the XEPs.

I doubt that modifications to the XEPs which replicate things the community already has figured out in a more consensual way would be accepted

I think it's also not taking advantage of community expertise that exists

that, too

What exactly things do we replicate?

Andrew Nenakhov, the use of XEP-0221 over XEP-0385 is a good example

but also the multi-user-chat discussion.

Why did community develop unnecessary standard instead of slightly updating 0221?

because data forms are---to me at least, but I think others may agree---not primarily a way to share (meta-)data but for interactive things. '221 was written with captchas in mind. '385 was specifically written with file sharing in mind, and I think it achieves that much better. There’s no way to annotate individual URLs in a <body/> using '221. '385 re-uses references for taht.

And with mix, I'm probably one of the few who mostly read it along with my dev team. It's broken by design. And we already have a working group chat that works great, it's fast and reliable. Maybe 30% work left on server side.

(but this is just waht I assume, '385 happenend before I was around here)

Let's not talk about MIX

Ok. )

I understand your point about code-first, standards-second, but I don’t see how a quick "Hey, we’re going to do file sharing with metadata and plan to use '221 for that, any reasons we shouldn’t?" email would’ve hurt anyone.

I think you would have gotten a clear response that building on '385 (I’m not saying that '385 is perfect, it can certainly use some love) would be preferrable.

You see. I respect you guys who do stuff . And I like the idea of federated messaging. I firmly believe humanity needs an independent protocol to exchange messages. That's why I'm using my time, money and attract investors to fund our efforts in this space. However, xmpp is already older than one of my developers who joined the team yesterday. If in all these years doing things like it's done xmpp didn't reach even a fraction of WhatsApp success, did it occur to you that things should be done differently? At least, tried to be done differently?

I see your point, however, going the same way WhatsApp (originally XMPP based, too) went by forking the protocol and ignoring the community won’t help here.

jonasw, ok I'll write an email on 221 use. We actually considered 385 and rejected it, I'll explain why.

that’d be great

Looking forward to see why you came to the conclusion that modifying '221 was a better way forward than modifying '385.

Andrew Nenakhov: we are building a protocol, not a product. You can't attract users with a protocol ;)

jonasw, we are not going the exact WhatsApp way. If we do stuff we'll publish protocol extensions and release open source products that support them.

(especially given that '385 is Experimental and thus has a much lower barrier to modification than '221 has)

Btw I'm still not over how xhtml XEP was made deprecated.

I’m still not certain on that one either.

I’m swaying between "it’s not *that* hard to get it right, even if done by simply disallowing all CSS" and "ugh. XHTML brings us a horde of security issues"

Disallowing all CSS isn’t even a good solution, the subset defined in 0071 is known to not have any vulnerability on current browsers.

Link Mauve, sure, but it’s not trivial to properly sanitise CSS

which is why a simple implementation may choose to ignore it altogether.

also, you’d want to modify any colors used to blend in with your UI.

Maybe I should split out xmpp-parsers’s XHTML handling into another crate, and provide bindings and compilation targets for a bunch of other languages.

jonasw, sure, and you can.

yes, but it’s work

work which can be gotten wrong

and which can lead to interesting issues when gotten wrong

although CSS will most likely only cause tracking since javascript URLs have been forbidden there. although I wouldn’t put my money on that one.

> As the team chat market is overcrowded by copycats, Nayego is different. Said yet another xmpp client developer.

Someone from the XMPP community was involved in it, is it?

I'd say nyco

Ge0rG, Team chat market isn't as crowded as it was yeterday, mind.

Dave Cridland: it's crowded by the bodies of those who tried and zombies of those who don't want to realize that they failed.

Ge0rG, Yeah, I was making a nod toward Stride/Hipchat/Jitsi.

Oh, I didn't know Jitsi was Atlassian as well. Is it also EOL now?

Jitsi was part of HipChat's business unit. The code was incorporated into Stride. Slack, OTOH, use a modified Janus SFU and things - I can't see they'd use much from it.

Will they make use of the team?

I wouldn't know.

I've not examined the details of the transfer, but I believe it's a sale of customers basically.

Except it's not providing customers with a clean migration path, except "install slack, kill your old chat".

Basically.

And honouring contracts, of course, though they were priced more or less identically.

I suppose this merger will make few people happy and many people sad, for a variety of reasons.

https://www.atlassian.com/blog/announcements/new-atlassian-slack-partnership

:'-(

Alex, Indeed. Nothing on Jitsi, mind, but that's part of the same business unit.

goffi, could you make a PR against '384 regarding the singleton implementation note?

There's a thing

Zash, that thing? https://xmpp.org/extensions/xep-0060.html#impl-singleton

https://xmpp.org/extensions/xep-0060.html#impl-singleton > When this pattern is desired, it is RECOMMENDED for the publisher to specify an ItemID of "current" to ensure that the publication of a new item will overwrite the existing item.

that’s what I’m talking about

jonasw: Yes. That thing.

jonasw: OK I can check that, will do it later this week-end.

goffi, thx!

With regards to CSI, does RFC6120#section-10.1 forbid the queuing of some stanzas, while letting other ones through?

I've been told it's fine to break RFC rules if the XEP allows it.

That's ... Surprising

And feels wrong

The rationale was that the rule breakage is negotiated and hence won't lead to trouble.

The problem I see is implementation-wise. If you encoded those rules in the lower layers, you must change those layers to implement an extension.

At best, I feel that that should be worded better in the XEP.

It now appears to describe that you should be compliant with section 10.1

MattJ, thoughs?

(penny for your)

My thoughts are that the XEP does not give any mandate to break the RFC

In-order processing is relied upon in multiple edge cases

The CSI XEP originally, very intentionally, avoided saying anything about what a server might do with its knowledge of the client state

And then some got annoyed because they couldn't be sure what the server would do.

Right, so community consensus seemed to be that it should at least provide a list of example behaviours

So you feel that you cannot withold say a status update from a client when you push it a message that was sent to it later?

I don't, no

> The CSI XEP originally, very intentionally, avoided saying anything about what a server might do with its knowledge of the client state The problem I see with that is that the client can't rely on anything once sending <inactive/>.

And would it be ok to withold from Alice a status update sent from Bob, when Bob later sends a message to Jacky?

Guus, I think it's a little safer between two distinct (bare?) JIDs

Holger, I don't see a problem with that

Sure, but it either does or doesn't break spec.

Holger, the CSI XEP was never intended to have any observable effect to the client, at a high level

Guus, CSI does not break the RFC

CSI is a way for the client to say whether it is "active" or not, it's very simple

The withold bit in my example, I mean

As far as I'm concerned, what servers do with this info is up to them

and not standardised

and neither should be, without very careful consideration

As far as I'm concerned, if the server breaks the client by messing around with the stream, the server is broken

Maybe someone feels like making a standard for this, possibly with an opt-in for the client, they are free to

So, as far as queuing goes, a CSI based optimization should not queue anything when sending something else to the same bare jid?

MattJ: Server implementors would have to anticipate all possible use cases to judge whether a given optimization is 'safe', i.e. "has no observable effect".

> So, as far as queuing goes, a CSI based optimization should not queue anything when sending something else to the same bare jid? In ejabberd, if I send you a message, I send all queued traffic from the same sender first. (Not sure why I don't just flush the queue in that case, the radio is woken anyway.)

From the same sender?

I'd think you'd flush all data to the same recipient. Possibly all its full jids?

Yeah, that's the part I'm not sure makes sense 🙂 (It's been a while and I recently thought about just flushing the queue whenever I send traffic.)

The thing I personally use has a single queue, that gets flushed when it's "full" or when there's something "important". The original order is preserved.

Guus: Right now if I send you a message from=alice, I only flush all stanzas that were sent from=alice.

Zash: I'm leaning towards something like that now too.

Yes, I'm planning to include the single-queue one in the next Prosody release

Holger: that's weird to me, as a message from Bob that was sent to the same recipient might arrive out of order.

The others are just too risky, but people can use them if they want (from prosody-modules) (and they do)

Despite no tests as to how effective any of these methods are on real traffic

Apart from that I try to dedup "state" things (presence, chat states, PEP), i.e. you only receive the current state. Mainly to avoid frequent queue flushes due to the queue size limit being exceeded.

Guus: The Mobile Optimizations XEP (or whatsitcalled, by Dave Cridland) basically says that too, "send a lot while the radio is hot"

Zash: makes sense

Guus: Yes queued traffic can go out of order that way. I.e. Alice sent presence before Bob, then Bob sent a message, so you receive Bob's presence before Alice's. This stuff can be hairy with MUC joins, for example. Just don't do this 🙂

Holger: As I've said before, the rules for things like that quickly grow complicated

I'm now wondering if I should stop CSI support with dropping chat state notifications and the like only...

Yes. But you might consider the complexity with it to avoid frequent queue flushes.

And not have a queue at all

*worth it

Dropping chat states -> never drop "active", never drop if there is any other payload in the message (that isn't recognised as safe to drop)

Is that the only required logic?

Why not 'active' if there's no other payload?

Because then someone could be typing, and then stop typing while I was inactive

and then they would be forever typing

in my client

I now have "drop messages that have no other child elements that do define a namespace, other than chatstates"

So you would have the perpetual typing issue?

Yeah, I didn't think of that

Isn't what Holger says to keep the last one?

That makes sense

Like that module that does that for presence already

But, really, only dropping chatstates, really doesn't make a dent in the traffic

Zash: yeah, but that'd involve queuing which I think either is flushed all the time (rendering CSI effectless) or break RFC mandated ordering

I do think trying to group stuff into batches is better than trying to drop things

If we bring back Compression-except-safe, then it should be possible to squeeze in all them chatstates without using much data

Meh, this all is a bit disappointing. I kinda hoped for a silver bullet.

Don't we all? :)

But when was the last time anyone measured XMPP client traffic or battery usage?

Sooooo.... Would things be more effective if we explicitly let the client tell the server for what kind of data it allows the server to break the delivery order requirement?

I don't think that's a nice thing to do to client devs

"only send me the last non subscription presence update for each contact that changed state after I went inactive"

Complex protocol for something where basically everyone will want the same rules

Well, without that, they don't get any benefit from CSI.

Just the rules aren't easy to describe or discover

Single queue has benefit

In theory

I'm not reading up for all the context, but ...

Is there any way we can get rid of CSI and instead disconnect when in the background and then get our reconnects to a sensible speed?

I'm not sure how different that is in the end

E.g. if the server optimizes the 198 queue with the same rules

I don't know. I just can't shake the feeling that both CSI and 198 Resumption are trying to work around a fundamental problem rather than address it.

Well, without SM you'd not show as online

What if they are addressing it?

Sorry, "available"

The issue being that getting current state is slow unless you were online when it changed, so we're trying to avoid going offline, with 198 letting us resume, and CSI letting us cut down the cost of being online.

But if we could solve the fundamental "It's expensive to catch up" issue, neither would be needed.

Kev: there's truth in that. Unsure if it's sufficiently feasible.

I think there's value in staying connected

If it's not feasible I think we end up with significant issues anyway, because at some point you're going to need to fire up a new client, or an old one will get disconnected beyond 198 periods, or whatever.

It feels to me like we should look at what we really want to do on login (rather than what current protocol dictates we do), and see how close to that we can get.