XSF Discussion - 2021-09-08

Google and Cisco at least used to participate

We certainly should care about those implementations, yes. We can’t write specs and then not care about breaking them in spec updates just because we didn’t know about them.

At Pando we (deliberately) stored groupchat messages in the personal archive for resynchronization etc. However, these weren't XEP-0045, but ESL's "Muclight", so there weren't any "holes" in that archive.

Kev, but should we care about implementations which don't participate in the standards process?

jonas’, I would argue we're unlikely to get them to use XMPP at all if it's unstable for their purposes.

jonas’, So hardly likely to help them decide to participate.

Kev, dwd: I've written down my questions on list and I'd really appreciate your input

Ge0rG, For XEP-0313? OK.

jonas’: Yes, we absolutely should. This isn’t an old boys club where we write specs only for ourselves.

dwd, what's the benefit for XMPP? implementations which use XMPP and then don't contribute to the standards process or anything, what does that give XMPP as a whole?

jonas’: Implementations.

what's the benefit of that?

What’s the point of a protocol without implementations?

I'm losing track of the meta discussion

Kev, well, fair enough I guess

jonas’, It seems to me we're more likely to see an increase in active participation in the community with more usage of the protocols. Equally, if we focus solely on existing participants and ignore the wider usage, I don't see that we can have any outcome bu a reduction in usage, and therefore participation.

I guess the question is: what is the likelihood that any current longstanding non-participant becomes a participant in the future? vs. the likelihood that XMPP loses ever more relevance because the wide-spread public use-cases cannot be covered.

I think that’s a false dichotomy.

or can only be covered by documents having an ugly EXPERIMENTAL tag slapped all over them because we cannot reach consensus.

There’s a world of difference between making unflagged deliberately breaking changes because we deem those implementations they break ‘not ones we care about’ and not addressing use cases. ✏

not having MAM stable is arguably "not addressing a use case"

jonas’, Well, yes, the experimental and lack of progress is certainly a problem.

Again, I think it’s a false dichotomy to say that the only two options are “stop it being stable” and “mark existing compliant implementations non-compliant (and therefore expect them not to interop)”.

Kev, enlighten me how to fix it other than that :)

Not introduce “SHOULD NOT” (or worse MUST NOT) on text that previously was SHOULD.

Job done.

It seems that at least one council member is not going to let that pass though.

(for good reasons for the wide-spread public usecases if you'd ask me)

Into meetings, AFK a few hours.

same, gl

I'm reading that as "Experimental is Implicit Stable, because it might have implementations that we don't know about and that would break if we change it"

And it's tangentially related to "we must not bump versions because the implementation ecosystem is so slow that we would fragment it"

Ge0rG, Well, yes and no. We protect deployed use of Experimental by namespace versioning, and that allows for parallel deployment of different versions of unstable protocols.

(which is an implementation burden much of the IM ecosystem cannot bear)

Ge0rG, But protocols that end up "stuck" in Experimental end up de-facto Stable. When de-facto is not reflected by the standards process it's not the de-facto that's wrong.

having a SHOULD written there when there are hundreds of deployments which do not honour it feels wrong though.

dwd: many implementations struggle with fully implementing a single version of a given protocol, and we have almost no visibility into which versions are supported where

jonas’, That, too.

Ge0rG, And also that.

I remember a discussion with a developer of a new client who was confused by that exact SHOULD, and I had to explain that the known server implementations don't even honor it.

Ge0rG, At least some that presumably you've heard of have it as an option. MongooseIM, for example.

Ge0rG, I have replied to your note with a possible path forward, anyway.

dwd: great, thanks!

dwd: Presumably you also want a feature advertised saying that such an option is present, yes?

Kev, That'd be sensible, yes.

I might make that change now while I have two minutes then, if it unblocks this.

I'm even more confused now.

dwd: I've responded on list now.

About what? Dave suggested adding a flag for whether you want gc results for your query or not. Absent that flag both behaviours are valid, meaning existing implementations that either do what’s written or don’t include them are all compliant, and that going forward we can understand what we’ll receive, which seems to address your concern.

I was implicitly assuming that type=groupchat equals MUC, and I think this is the cause of most of the confusion and debate.

Kev: I don't like it, because not having a default value is bad protocol, but I'm willing to take it as the lesser evil compared to namespace-bumping or eternal-experimental. Something is going to die in me when I +0 or +1 that.

However, I'm still unconvinced that storing MUC history in the personal archive and returning it without an explicit query is a good idea.

The shattering of your immortal soul aside, if I put some security considerations in, and add text/query parameter/disco features along Dave’s lines, you’ll withdraw your imminent -1?

Kev: is it a matter of rushing it through the Council today, or can we maybe have a few more days of pondering about a less unreasonable alternative?

I don’t particularly care about Experimental->Draft^h^h^h^h^hStable for this at the moment, but I know lots of other people do.

Ge0rG: I leave when Council do stuff up to Council, I just want to remove your current objections.

This is a discussion that I wish we would have had during the LC.

Kev: I think that it would be better to exchange a few more mails and get me to a solid +1 instead of getting me to a +0 today with a solution that's cementing a bad protocol design.

If your idea of a solid +1 is one in which it is forbidden to send groupchat results, as some current deployments do, without bumping namespaces, I don’t think we can achieve that while maintaining the (implicit?) promise that when we make breaking changes to Experimental XEPs we do something with namespaces or features such that compliant deployments don’t become non-compliant.

Kev: but it'd be great if you could suggest a security section mentioning that a client MUST filter by query-id and by from address

oh wait, that would be a breaking change as well.

FWIW, it’s perfectly fine for a client to only filter on from address if it serialises queries, from a security considerations perspective.

Kev: from my reading of the current 0313 text, a client can not rely on a server actually delivering type=groupchat results.

This is of course different with servers that will nevertheless send those.

> Kev: from my reading of the current 0313 text, a client can not rely on a server actually delivering type=groupchat results. Indeed, but it must be able to cope with receiving them. If the XEP were changed to say “You won’t get them”, servers that do send them could break clients that know they won’t receive them.

Kev: right.

Can we add a feature and a query element with a default value and have a note that the default is undefined when the feature is not present?

ie. no feature -> unknown; feature -> default=no-groupchat?

<p>Servers that understand the 'include-groupchat' field MUST advertise the 'urn:xmpp:mam:2#gc-field' (even if they cannot return groupchat messages), and servers that understand the 'include-groupchat' field and store groupchat messages in the user's archive must advertise the 'urn:xmpp:mam:2#groupchat-available' feature</p>

That’s what I propose for disco.

So that means if you don’t store gc, you advertise #gc-field. If you store gc you advertise #gc-field and #gc-available.

Kev: nitpick: rename the feature to `urn:xmpp:mam:2#groupchat-field` or maybe `urn:xmpp:mam:2#include-groupchat`

I did the opposite and changed both to gc.

to make it better searchable for "groupchat"

👍

But I can revert, this is not a death hill.

I have a slight bias towards `groupchat` and a strong bias towards consistency

now how will that field be used? should it always be present in the query, with a boolean value for returning / not returning?

So if I understand your request, when #groupchat-field is advertised by the server, you’re saying that clients that don’t specify the gc field shouldn’t get groupchat?

Kev: yes

I know of deployments where that would be unfortunate.

it sounds like a breaking change when you say it

Where if the server was upgraded to a new MAM, but the client wasn’t, behaviour would change in an unfortunate way.

I think it would be better to go with Dave’s suggestion and leave the behaviour undefined in the face of not specifying the field, because then servers can leave their existing behaviour in place for undefined, and existing deployments don’t change behaviour on upgrade.

I dislike it a bit less now.

(I do agree that it is at best unsatisfying to have optional options that don’t have defaults, but …)

The alternative route would require a namespace bump *and* a feature

I think Dave’s proposal is the least bad option we’ve got.

Given where we are.

I still don't see how returning semi-random blocks of MUC history is of any use.

I think that’s because you’re viewing MUC history in the user archive as being ‘everything that happened’ and I view it as ‘everything the user received’.

But maybe you can elaborate that and the other questions on-list as well :)

Kev: even in the latter case it's going to be weird.

My primary use of MAM is searching for something that I know I saw somewhere, but I can’t remember the details of, nor whether it was in one of the team rooms or in a direct message.

Should it _also_ include the history that the client already received via live MUC traffic?

Should a client use MAM IDs from MUC messages for an `after-id` query?

Yes, I agree there is weirdness.

The existing implementations must have handled that weirdness in some way, right?

Kev: which protocol are you using for searching?

313 with a search field in the data form.

The `urn:example:xmpp:free-text-search` one? ;)

I think we predate that, but I don’t recall TBH.

I suppose that having a "free text search" field standardized in 313 would be of value as well.

I'd argue that returning MUC results in queries _by default_ is a bad idea, but I can see how it makes sense for text-search queries. But those are only adding another option to the form, so making that an indicator to change the default response is weird as well.

<p>A client MUST verify the source of MAM query results against an open query (i.e. checking the stanza 'from' matches the entity that was queried) and MUST either ignore or otherwise disregard (maybe with a warning to the user) unsolicited results. This is to avoid the situation where a malicious entity sends MAM results while the client is querying a different entity and the client processes the malicious results as if they were part of the legitimate results.</p>

That sort of thing is what you care about, right Ge0rG?

"Additionally, if the client has multiple queries in flight at once, it MUST also check that the query ID for a result matches that of an open query for that entity."

Kev: you make it sound like you may only receive MAM results while at least one query is running

I think that "unsolicited MAM results" would be the most probable incarnation of the attack, because they don't rely on race conditions

So it might be good to consolidate the pre-conditions (unsolicited, wrong from, wrong query-id) first, and then tell to ignore / warn the user?

and how are we going to introduce that MUST without namespace-bumping? This might even break existing implementations, if those are susceptible to impersonation attacks.

> Kev: you make it sound like you may only receive MAM results while at least one query is running That’s right, isn’t it? If you receive results outside a query, you must drop them.

> and how are we going to introduce that MUST without namespace-bumping? This might even break existing implementations, if those are susceptible to impersonation attacks. I think you’re being facetious here, but just in case, I don’t think closing this particular security vulnerability would be a reason to namespace bump.

Since anyone vulnerable to such an attack is already broken, whether by behaviour or spec.

> If you receive results outside a query, you must drop them. This may sound like an obvious thing, but many implementations have callbacks that are called on each message / each forwarded element and might just process those even outside of a query

So I'd rather write it down in that paragraph

"A client MUST verify the source of MAM query results against an open query (i.e. checking the stanza 'from' matches the entity that was queried) and MUST either ignore or otherwise disregard (maybe with a warning to the user) unsolicited results - whether because the 'from' doesn't match an open query, or because there is no open query"

Bit after -

Good enough for me :)

Kev: Also feel free to copy&paste the CVE-2019-16235 and CVE-2020-26547 blocks from 0280, as those also affect MAM

Jonas thought sorting out the CVEs was Editorial, so I’ll leave that for the moment.

Good enough.

I figure just getting an author patch in with the other comments is probably providing more urgentness right now.

Kev: you are right.

Regarding process, do we need another LC or can that be considered as "implementing LC feedback from last time"?

As far as I would care, this is addressing feedback issued since LC.

I know LCs are two weeks or whatever, but in practice from the moment LC is issued until it advances to Drable is one big LC period.

So we can actually vote on advancing it today.

If I get this patch submitted and an Editor gets it merged (or Council vote based on the merging), potentially. I could be dragged away from it at any moment, though, and I’m pretty slow at writing XEP stuff.

Writing XEP text is not an easy job.

Ok, please be kind about this paragraph, because I don’t think it’s straightforward to write (I’ll include the specifics of ‘include-groupchat’ else where in the protocol section. This is in business rules: <p>Previous versions of this specification stated that a server SHOULD also include messages of type 'groupchat' that have a &lt;body&gt;. This advice has now been dropped, and servers MAY include groupchat messages in their archives. Whether a server stores groupchat messages or not is now left as an implementation (or deployment) decision. Whether a client wants to receive groupchat messages in results can be signalled with the 'include-groupchat' field (if supported by the server - see "Determining support") - where the server doesn't support this field, or where a client doesn't specify it in the query, whether groupchat messages are included in the result is implementation-defined; this allows existing deployments to not break with the introduction of the 'include-groupchat' query field in a later version of this specification, but it is RECOMMENDED that all client implementations of the current version of this specification always include the field where the server supports it, and that servers support it.</p>

Ge0rG & dwd: Does that roughly match your expectations?

SGTM

Kev: +1

would anything break if a client unconditionally includes <include-groupchat>?

Looks good from my side.

Ge0rG, I did wonder about that. I think forms generally pre-suppose ignoring unknown fields, but I'd need to check.

Ge0rG: Depends how the server processes unexpected query fields. I wouldn’t like to recommend it.

Ge0rG, possibly.

Extensible.

I think the question "what happens on undeclared fields" was raised in the previous LC

Extensible doesn’t mean that when the server says “You can use these fields” that it should accept other ones :)

Kev: well, normally it should mean "ignore all the fields you don't know"

jonas’: indeed, because the implication was to return results that don't match what the client asked for.

Actually, including it serves no purpose.

Because if the server doesn’t support it, you won’t get it respecting it. ✏

So even if you could always include gc=false, you’d still have to expect gc results.

Kev: the purpose is for a client that knows what it wants to always tell the server, without requiring a feature discovery round-trip

Kev: well, you also have to ignore those results from a server that doesn't have the feature.

Regardless - if we later determine that it is safe to include it’s a non-breaking change to a Drable (or Final) XEP to say “A client can safely always include this feature”.

But saying it now and later finding it was wrong is a breaking change.

Rrrrright.

So I suggest we don’t add it now.

<p>If a client includes a form field that the server does not recognise, the server MUST respond with a 'feature-not-implemented' error.</p>

So it’s definitely not safe.

Yay.

so s/where the server supports it/iff the server supports it/?

We have so far exactly one use of "iff", in 0390

Ok, another please be kind;

<section3 topic='Including groupchat results in a user archive' anchor='query-include-groupchat'> <p>If the server advertises that it includes groupchat messages in a user's archive (see <link url='#support'>Determining support</link>), a client may query a user archive and request for them to be included in the result with the 'include-groupchat' field set to 'true'. </p> <example caption='Querying the archive and including groupchat messages in results'><![CDATA[ <iq type='set' id='juliet1'> <query xmlns='urn:xmpp:mam:2'> <x xmlns='jabber:x:data' type='submit'> <field var='FORM_TYPE' type='hidden'> <value>urn:xmpp:mam:2</value> </field> <field var='include-groupchat'> <value>true</value> </field> ... </x> </query> </iq> ]]></example> <p>If the server advertises that it includes groupchat messages in the archive, or it advertises that it doesn't, a client may request that they not be included by setting the 'include-groupchat' field to 'false'.</p> <example caption='Querying the archive and excluding groupchat messages from results'><![CDATA[ <iq type='set' id='juliet1'> <query xmlns='urn:xmpp:mam:2'> <x xmlns='jabber:x:data' type='submit'> <field var='FORM_TYPE' type='hidden'> <value>urn:xmpp:mam:2</value> </field> <field var='include-groupchat'> <value>false</value> </field> ... </x> </query> </iq> ]]></example> <p>Note that where the client doesn't specify the 'include-groupchat' field, it is implementation-defined whether groupchat messages are included in the results (see <link url='#business_rules'>Business Rules</link>).</p> </section3>

Ge0rG, I came to the conclusion that `iff` is too close to a typo and it shoudl be written as `if and only if` always

jonas’: +1 to that

If I’m writing for an English-first audience, I’ll happily use iff. I think in XEPs it’s probably ill-advised.

Kev: you might want to add a note that if a client adds the field on a server not advertising the feature, it will receive a `feature-not-implemented` error

That can’t hurt.

"Clients MUST NOT include this field where servers don't advertise support, as the server would reject such a form."

Kev: +1

Ok, I think I have a suitable change, then.

Ge0rG (and others) - would you mind checking https://github.com/xsf/xeps/compare/master...Kev:xep313/LC-feedback-Ge0rG?expand=1 is close enough to what you expect, please, before I open the PR? I’m running out of morning and I increasingly urgently need to move on to other things :)

Kev: that will only open a PR window, ITYM https://github.com/Kev/xeps/commit/992d40b20995692b1d3db5a760948d88103283ef

It shows a diff for me, but probably :)

Kev: could you add a <section2 topic='Sender Impersonation' anchor='security-impersonation'> around the security paragraph?

other than that, it looks great to me!

+ </section2> + <section2 topic='Sender Impersonation' anchor='security-impersonation'>

Branch updated. I’ll submit, thanks.

I still think that we'd greatly benefit from a statement like "Servers SHOULD NOT send offline history to clients that perform a MAM query before their initial presence"

I don’t disagree in principle, although I kinda feel it’s too big a can of worms for right now.

And we obviously can’t fully SHOULD NOT without some sort of namespace guarding.

Well, this is exactly the can of worms you want to address before going Drable.

I can pop in a line to business rules if you’re happy that it’s non-normative.

This is exactly the sort of thing I'd like discussed among server and client developers.

<p>A server MAY choose not to deliver offline messages to a client that has already queried their MAM archive and received the archived copies of those messages that would otherwise be delivered - while not required of an implementation, this is helpful to avoid duplicate messages for clients, so is suggested.</p> Seems safe enough, to me.

Kev: great wording, +1

Also pushed.

I’ll raise the PR, then.

Kev: please do!

I think that’s done now - https://github.com/xsf/xeps/pull/1104

I guess I *could* hit merge myself, but I’d probably better leave it for jonas’ in case I messed something up in the PR.

yes please don't just hit merge

we don't have any automation there and it'll confuse the heck out of me

mew

So I wanted to have a perma-link to "this year's compliance suite XEP" and it's not possible :(

Ge0rG, nginx redirect?

wurstsalat: something like https://xmpp.org/compliancesuite/ maybe?

wurstsalat: who can create such a redirect?

Ge0rG, this file (I think) https://github.com/xsf/xmpp.org/blob/master/deploy/xsf.conf

wurstsalat: I'm not sure I should dare doing that without asking the A-Team, sorry, the i-team first.

Ge0rG, sure! those redirects work when building the page via docker

https://github.com/xsf/xmpp.org/pull/960 is for the i-team

please dont merge untill we have the new website setup

emus: better comment that on the PR

yep

MattJ, Zash: Do you guys have a list which clients support the technically non-standard invite-based registrations you implemented? (i.e. prosody mod_invites)

making some progress with dynamically built setup guides for my services' site and need that info to recommend people how to do things depending on what OS they're on and what client i'm recommending per OS. :)

phryk, I recently learned that it was 99% standardized as https://xmpp.org/extensions/xep-0445.html

but it had somehow passed me by

If I recall the situation right you and Zash implemented that but it wasn't complete, so you implemented the necessary extra bits and added them to the XEP but the original author reverted those changes to the XEP.

also none of the known client doaps report support for it :P