XSF Discussion - 2021-09-08

  1. MattJ

    Google and Cisco at least used to participate

  2. Kev

    We certainly should care about those implementations, yes. We can’t write specs and then not care about breaking them in spec updates just because we didn’t know about them.

  3. dwd

    At Pando we (deliberately) stored groupchat messages in the personal archive for resynchronization etc. However, these weren't XEP-0045, but ESL's "Muclight", so there weren't any "holes" in that archive.

  4. jonas’

    Kev, but should we care about implementations which don't participate in the standards process?

  5. dwd

    jonas’, I would argue we're unlikely to get them to use XMPP at all if it's unstable for their purposes.

  6. dwd

    jonas’, So hardly likely to help them decide to participate.

  7. Ge0rG

    Kev, dwd: I've written down my questions on list and I'd really appreciate your input

  8. dwd

    Ge0rG, For XEP-0313? OK.

  9. Kev

    jonas’: Yes, we absolutely should. This isn’t an old boys club where we write specs only for ourselves.

  10. jonas’

    dwd, what's the benefit for XMPP? implementations which use XMPP and then don't contribute to the standards process or anything, what does that give XMPP as a whole?

  11. Kev

    jonas’: Implementations.

  12. jonas’

    what's the benefit of that?

  13. Kev

    What’s the point of a protocol without implementations?

  14. Ge0rG

    I'm losing track of the meta discussion

  15. jonas’

    Kev, well, fair enough I guess

  16. dwd

    jonas’, It seems to me we're more likely to see an increase in active participation in the community with more usage of the protocols. Equally, if we focus solely on existing participants and ignore the wider usage, I don't see that we can have any outcome bu a reduction in usage, and therefore participation.

  17. jonas’

    I guess the question is: what is the likelihood that any current longstanding non-participant becomes a participant in the future? vs. the likelihood that XMPP loses ever more relevance because the wide-spread public use-cases cannot be covered.

  18. Kev

    I think that’s a false dichotomy.

  19. jonas’

    or can only be covered by documents having an ugly EXPERIMENTAL tag slapped all over them because we cannot reach consensus.

  20. Kev

    There’s a world of difference between not making unflagged deliberately breaking changes because we need those implementations they break ‘not ones we care about’ and not addressing use cases.

  21. Kev

    There’s a world of difference between making unflagged deliberately breaking changes because we need those implementations they break ‘not ones we care about’ and not addressing use cases.

  22. Kev

    There’s a world of difference between making unflagged deliberately breaking changes because we deem those implementations they break ‘not ones we care about’ and not addressing use cases.

  23. jonas’

    not having MAM stable is arguably "not addressing a use case"

  24. dwd

    jonas’, Well, yes, the experimental and lack of progress is certainly a problem.

  25. Kev

    Again, I think it’s a false dichotomy to say that the only two options are “stop it being stable” and “mark existing compliant implementations non-compliant (and therefore expect them not to interop)”.

  26. jonas’

    Kev, enlighten me how to fix it other than that :)

  27. Kev

    Not introduce “SHOULD NOT” (or worse MUST NOT) on text that previously was SHOULD.

  28. Kev

    Job done.

  29. jonas’

    It seems that at least one council member is not going to let that pass though.

  30. jonas’

    (for good reasons for the wide-spread public usecases if you'd ask me)

  31. Kev

    Into meetings, AFK a few hours.

  32. jonas’

    same, gl

  33. Ge0rG

    I'm reading that as "Experimental is Implicit Stable, because it might have implementations that we don't know about and that would break if we change it"

  34. Ge0rG

    And it's tangentially related to "we must not bump versions because the implementation ecosystem is so slow that we would fragment it"

  35. dwd

    Ge0rG, Well, yes and no. We protect deployed use of Experimental by namespace versioning, and that allows for parallel deployment of different versions of unstable protocols.

  36. jonas’

    (which is an implementation burden much of the IM ecosystem cannot bear)

  37. dwd

    Ge0rG, But protocols that end up "stuck" in Experimental end up de-facto Stable. When de-facto is not reflected by the standards process it's not the de-facto that's wrong.

  38. jonas’

    having a SHOULD written there when there are hundreds of deployments which do not honour it feels wrong though.

  39. Ge0rG

    dwd: many implementations struggle with fully implementing a single version of a given protocol, and we have almost no visibility into which versions are supported where

  40. dwd

    jonas’, That, too.

  41. dwd

    Ge0rG, And also that.

  42. Ge0rG

    I remember a discussion with a developer of a new client who was confused by that exact SHOULD, and I had to explain that the known server implementations don't even honor it.

  43. dwd

    Ge0rG, At least some that presumably you've heard of have it as an option. MongooseIM, for example.

  44. dwd

    Ge0rG, I have replied to your note with a possible path forward, anyway.

  45. Ge0rG

    dwd: great, thanks!

  46. Kev

    dwd: Presumably you also want a feature advertised saying that such an option is present, yes?

  47. dwd

    Kev, That'd be sensible, yes.

  48. Kev

    I might make that change now while I have two minutes then, if it unblocks this.

  49. Ge0rG

    I'm even more confused now.

  50. Ge0rG

    dwd: I've responded on list now.

  51. Kev

    About what? Dave suggested adding a flag for whether you want gc results for your query or not. Absent that flag both behaviours are valid, meaning existing implementations that either do what’s written or don’t include them are all compliant, and that going forward we can understand what we’ll receive, which seems to address your concern.

  52. Ge0rG

    I was implicitly assuming that type=groupchat equals MUC, and I think this is the cause of most of the confusion and debate.

  53. Ge0rG

    Kev: I don't like it, because not having a default value is bad protocol, but I'm willing to take it as the lesser evil compared to namespace-bumping or eternal-experimental. Something is going to die in me when I +0 or +1 that.

  54. Ge0rG

    However, I'm still unconvinced that storing MUC history in the personal archive and returning it without an explicit query is a good idea.

  55. Kev

    The shattering of your immortal soul aside, if I put some security considerations in, and add text/query parameter/disco features along Dave’s lines, you’ll withdraw your imminent -1?

  56. Ge0rG

    Kev: is it a matter of rushing it through the Council today, or can we maybe have a few more days of pondering about a less unreasonable alternative?

  57. Kev

    I don’t particularly care about Experimental->Draft^h^h^h^h^hStable for this at the moment, but I know lots of other people do.

  58. Kev

    Ge0rG: I leave when Council do stuff up to Council, I just want to remove your current objections.

  59. Ge0rG

    This is a discussion that I wish we would have had during the LC.

  60. Ge0rG

    Kev: I think that it would be better to exchange a few more mails and get me to a solid +1 instead of getting me to a +0 today with a solution that's cementing a bad protocol design.

  61. Kev

    If your idea of a solid +1 is one in which it is forbidden to send groupchat results, as some current deployments do, without bumping namespaces, I don’t think we can achieve that while maintaining the (implicit?) promise that when we make breaking changes to Experimental XEPs we do something with namespaces or features such that compliant deployments don’t become non-compliant.

  62. Ge0rG

    Kev: but it'd be great if you could suggest a security section mentioning that a client MUST filter by query-id and by from address

  63. Ge0rG

    oh wait, that would be a breaking change as well.

  64. Kev

    FWIW, it’s perfectly fine for a client to only filter on from address if it serialises queries, from a security considerations perspective.

  65. Ge0rG

    Kev: from my reading of the current 0313 text, a client can not rely on a server actually delivering type=groupchat results.

  66. Ge0rG

    This is of course different with servers that will nevertheless send those.

  67. Kev

    > Kev: from my reading of the current 0313 text, a client can not rely on a server actually delivering type=groupchat results. Indeed, but it must be able to cope with receiving them. If the XEP were changed to say “You won’t get them”, servers that do send them could break clients that know they won’t receive them.

  68. Ge0rG

    Kev: right.

  69. Ge0rG

    Can we add a feature and a query element with a default value and have a note that the default is undefined when the feature is not present?

  70. Ge0rG

    ie. no feature -> unknown; feature -> default=no-groupchat?

  71. Kev

    <p>Servers that understand the 'include-groupchat' field MUST advertise the 'urn:xmpp:mam:2#gc-field' (even if they cannot return groupchat messages), and servers that understand the 'include-groupchat' field and store groupchat messages in the user's archive must advertise the 'urn:xmpp:mam:2#groupchat-available' feature</p>

  72. Kev

    That’s what I propose for disco.

  73. Kev

    So that means if you don’t store gc, you advertise #gc-field. If you store gc you advertise #gc-field and #gc-available.

  74. Ge0rG

    Kev: nitpick: rename the feature to `urn:xmpp:mam:2#groupchat-field` or maybe `urn:xmpp:mam:2#include-groupchat`

  75. Kev

    I did the opposite and changed both to gc.

  76. Ge0rG

    to make it better searchable for "groupchat"

  77. Ge0rG


  78. Kev

    But I can revert, this is not a death hill.

  79. Ge0rG

    I have a slight bias towards `groupchat` and a strong bias towards consistency

  80. Ge0rG

    now how will that field be used? should it always be present in the query, with a boolean value for returning / not returning?

  81. Kev

    So if I understand your request, when #groupchat-field is advertised by the server, you’re saying that clients that don’t specify the gc field shouldn’t get groupchat?

  82. Ge0rG

    Kev: yes

  83. Kev

    I know of deployments where that would be unfortunate.

  84. Ge0rG

    it sounds like a breaking change when you say it

  85. Kev

    Where if the server was upgraded to a new MAM, but the client wasn’t, behaviour would change in an unfortunate way.

  86. Kev

    I think it would be better to go with Dave’s suggestion and leave the behaviour undefined in the face of not specifying the field, because then servers can leave their existing behaviour in place for undefined, and existing deployments don’t change behaviour on upgrade.

  87. Ge0rG

    I dislike it a bit less now.

  88. Kev

    (I do agree that it is at best unsatisfying to have optional options that don’t have defaults, but …)

  89. Ge0rG

    The alternative route would require a namespace bump *and* a feature

  90. Kev

    I think Dave’s proposal is the least bad option we’ve got.

  91. Kev

    Given where we are.

  92. Ge0rG

    I still don't see how returning semi-random blocks of MUC history is of any use.

  93. Kev

    I think that’s because you’re viewing MUC history in the user archive as being ‘everything that happened’ and I view it as ‘everything the user received’.

  94. Ge0rG

    But maybe you can elaborate that and the other questions on-list as well :)

  95. Ge0rG

    Kev: even in the latter case it's going to be weird.

  96. Kev

    My primary use of MAM is searching for something that I know I saw somewhere, but I can’t remember the details of, nor whether it was in one of the team rooms or in a direct message.

  97. Ge0rG

    Should it _also_ include the history that the client already received via live MUC traffic?

  98. Ge0rG

    Should a client use MAM IDs from MUC messages for an `after-id` query?

  99. Kev

    Yes, I agree there is weirdness.

  100. Ge0rG

    The existing implementations must have handled that weirdness in some way, right?

  101. Ge0rG

    Kev: which protocol are you using for searching?

  102. Kev

    313 with a search field in the data form.

  103. Ge0rG

    The `urn:example:xmpp:free-text-search` one? ;)

  104. Kev

    I think we predate that, but I don’t recall TBH.

  105. Ge0rG

    I suppose that having a "free text search" field standardized in 313 would be of value as well.

  106. Ge0rG

    I'd argue that returning MUC results in queries _by default_ is a bad idea, but I can see how it makes sense for text-search queries. But those are only adding another option to the form, so making that an indicator to change the default response is weird as well.

  107. Kev

    <p>A client MUST verify the source of MAM query results against an open query (i.e. checking the stanza 'from' matches the entity that was queried) and MUST either ignore or otherwise disregard (maybe with a warning to the user) unsolicited results. This is to avoid the situation where a malicious entity sends MAM results while the client is querying a different entity and the client processes the malicious results as if they were part of the legitimate results.</p>

  108. Kev

    That sort of thing is what you care about, right Ge0rG?

  109. Kev

    "Additionally, if the client has multiple queries in flight at once, it MUST also check that the query ID for a result matches that of an open query for that entity."

  110. Ge0rG

    Kev: you make it sound like you may only receive MAM results while at least one query is running

  111. Ge0rG

    I think that "unsolicited MAM results" would be the most probable incarnation of the attack, because they don't rely on race conditions

  112. Ge0rG

    So it might be good to consolidate the pre-conditions (unsolicited, wrong from, wrong query-id) first, and then tell to ignore / warn the user?

  113. Ge0rG

    and how are we going to introduce that MUST without namespace-bumping? This might even break existing implementations, if those are susceptible to impersonation attacks.

  114. Kev

    > Kev: you make it sound like you may only receive MAM results while at least one query is running That’s right, isn’t it? If you receive results outside a query, you must drop them.

  115. Kev

    > and how are we going to introduce that MUST without namespace-bumping? This might even break existing implementations, if those are susceptible to impersonation attacks. I think you’re being facetious here, but just in case, I don’t think closing this particular security vulnerability would be a reason to namespace bump.

  116. Kev

    Since anyone vulnerable to such an attack is already broken, whether by behaviour or spec.

  117. Ge0rG

    > If you receive results outside a query, you must drop them. This may sound like an obvious thing, but many implementations have callbacks that are called on each message / each forwarded element and might just process those even outside of a query

  118. Ge0rG

    So I'd rather write it down in that paragraph

  119. Kev

    "A client MUST verify the source of MAM query results against an open query (i.e. checking the stanza 'from' matches the entity that was queried) and MUST either ignore or otherwise disregard (maybe with a warning to the user) unsolicited results - whether because the 'from' doesn't match an open query, or because there is no open query"

  120. Kev

    Bit after -

  121. Ge0rG

    Good enough for me :)

  122. Ge0rG

    Kev: Also feel free to copy&paste the CVE-2019-16235 and CVE-2020-26547 blocks from 0280, as those also affect MAM

  123. Kev

    Jonas thought sorting out the CVEs was Editorial, so I’ll leave that for the moment.

  124. Ge0rG

    Good enough.

  125. Kev

    I figure just getting an author patch in with the other comments is probably providing more urgentness right now.

  126. Ge0rG

    Kev: you are right.

  127. Ge0rG

    Regarding process, do we need another LC or can that be considered as "implementing LC feedback from last time"?

  128. Kev

    As far as I would care, this is addressing feedback issued since LC.

  129. Kev

    I know LCs are two weeks or whatever, but in practice from the moment LC is issued until it advances to Drable is one big LC period.

  130. Ge0rG

    So we can actually vote on advancing it today.

  131. Kev

    If I get this patch submitted and an Editor gets it merged (or Council vote based on the merging), potentially. I could be dragged away from it at any moment, though, and I’m pretty slow at writing XEP stuff.

  132. Ge0rG

    Writing XEP text is not an easy job.

  133. Kev

    Ok, please be kind about this paragraph, because I don’t think it’s straightforward to write (I’ll include the specifics of ‘include-groupchat’ else where in the protocol section. This is in business rules: <p>Previous versions of this specification stated that a server SHOULD also include messages of type 'groupchat' that have a &lt;body&gt;. This advice has now been dropped, and servers MAY include groupchat messages in their archives. Whether a server stores groupchat messages or not is now left as an implementation (or deployment) decision. Whether a client wants to receive groupchat messages in results can be signalled with the 'include-groupchat' field (if supported by the server - see "Determining support") - where the server doesn't support this field, or where a client doesn't specify it in the query, whether groupchat messages are included in the result is implementation-defined; this allows existing deployments to not break with the introduction of the 'include-groupchat' query field in a later version of this specification, but it is RECOMMENDED that all client implementations of the current version of this specification always include the field where the server supports it, and that servers support it.</p>

  134. Kev

    Ge0rG & dwd: Does that roughly match your expectations?

  135. jonas’


  136. Ge0rG

    Kev: +1

  137. Ge0rG

    would anything break if a client unconditionally includes <include-groupchat>?

  138. dwd

    Looks good from my side.

  139. dwd

    Ge0rG, I did wonder about that. I think forms generally pre-suppose ignoring unknown fields, but I'd need to check.

  140. Kev

    Ge0rG: Depends how the server processes unexpected query fields. I wouldn’t like to recommend it.

  141. jonas’

    Ge0rG, possibly.

  142. Ge0rG


  143. jonas’

    I think the question "what happens on undeclared fields" was raised in the previous LC

  144. Kev

    Extensible doesn’t mean that when the server says “You can use these fields” that it should accept other ones :)

  145. Ge0rG

    Kev: well, normally it should mean "ignore all the fields you don't know"

  146. Ge0rG

    jonas’: indeed, because the implication was to return results that don't match what the client asked for.

  147. Kev

    Actually, including it serves no purpose.

  148. Kev

    Because if the server doesn’t support it, you won’t get it respecting in.

  149. Kev

    Because if the server doesn’t support it, you won’t get it respecting it.

  150. Kev

    So even if you could always include gc=false, you’d still have to expect gc results.

  151. Ge0rG

    Kev: the purpose is for a client that knows what it wants to always tell the server, without requiring a feature discovery round-trip

  152. Ge0rG

    Kev: well, you also have to ignore those results from a server that doesn't have the feature.

  153. Kev

    Regardless - if we later determine that it is safe to include it’s a non-breaking change to a Drable (or Final) XEP to say “A client can safely always include this feature”.

  154. Kev

    But saying it now and later finding it was wrong is a breaking change.

  155. Ge0rG


  156. Kev

    So I suggest we don’t add it now.

  157. Kev

    <p>If a client includes a form field that the server does not recognise, the server MUST respond with a 'feature-not-implemented' error.</p>

  158. Kev

    So it’s definitely not safe.

  159. Ge0rG


  160. Ge0rG

    so s/where the server supports it/iff the server supports it/?

  161. Ge0rG

    We have so far exactly one use of "iff", in 0390

  162. Kev

    Ok, another please be kind;

  163. Kev

    <section3 topic='Including groupchat results in a user archive' anchor='query-include-groupchat'> <p>If the server advertises that it includes groupchat messages in a user's archive (see <link url='#support'>Determining support</link>), a client may query a user archive and request for them to be included in the result with the 'include-groupchat' field set to 'true'. </p> <example caption='Querying the archive and including groupchat messages in results'><![CDATA[ <iq type='set' id='juliet1'> <query xmlns='urn:xmpp:mam:2'> <x xmlns='jabber:x:data' type='submit'> <field var='FORM_TYPE' type='hidden'> <value>urn:xmpp:mam:2</value> </field> <field var='include-groupchat'> <value>true</value> </field> ... </x> </query> </iq> ]]></example> <p>If the server advertises that it includes groupchat messages in the archive, or it advertises that it doesn't, a client may request that they not be included by setting the 'include-groupchat' field to 'false'.</p> <example caption='Querying the archive and excluding groupchat messages from results'><![CDATA[ <iq type='set' id='juliet1'> <query xmlns='urn:xmpp:mam:2'> <x xmlns='jabber:x:data' type='submit'> <field var='FORM_TYPE' type='hidden'> <value>urn:xmpp:mam:2</value> </field> <field var='include-groupchat'> <value>false</value> </field> ... </x> </query> </iq> ]]></example> <p>Note that where the client doesn't specify the 'include-groupchat' field, it is implementation-defined whether groupchat messages are included in the results (see <link url='#business_rules'>Business Rules</link>).</p> </section3>

  164. jonas’

    Ge0rG, I came to the conclusion that `iff` is too close to a typo and it shoudl be written as `if and only if` always

  165. Ge0rG

    jonas’: +1 to that

  166. Kev

    If I’m writing for an English-first audience, I’ll happily use iff. I think in XEPs it’s probably ill-advised.

  167. Ge0rG

    Kev: you might want to add a note that if a client adds the field on a server not advertising the feature, it will receive a `feature-not-implemented` error

  168. Kev

    That can’t hurt.

  169. Kev

    "Clients MUST NOT include this field where servers don't advertise support, as the server would reject such a form."

  170. Ge0rG

    Kev: +1

  171. Kev

    Ok, I think I have a suitable change, then.

  172. Kev

    Ge0rG (and others) - would you mind checking https://github.com/xsf/xeps/compare/master...Kev:xep313/LC-feedback-Ge0rG?expand=1 is close enough to what you expect, please, before I open the PR? I’m running out of morning and I increasingly urgently need to move on to other things :)

  173. Ge0rG

    Kev: that will only open a PR window, ITYM https://github.com/Kev/xeps/commit/992d40b20995692b1d3db5a760948d88103283ef

  174. Kev

    It shows a diff for me, but probably :)

  175. Ge0rG

    Kev: could you add a <section2 topic='Sender Impersonation' anchor='security-impersonation'> around the security paragraph?

  176. Ge0rG

    other than that, it looks great to me!

  177. Kev

    + </section2> + <section2 topic='Sender Impersonation' anchor='security-impersonation'>

  178. Kev

    Branch updated. I’ll submit, thanks.

  179. Ge0rG

    I still think that we'd greatly benefit from a statement like "Servers SHOULD NOT send offline history to clients that perform a MAM query before their initial presence"

  180. Kev

    I don’t disagree in principle, although I kinda feel it’s too big a can of worms for right now.

  181. Kev

    And we obviously can’t fully SHOULD NOT without some sort of namespace guarding.

  182. Ge0rG

    Well, this is exactly the can of worms you want to address before going Drable.

  183. Kev

    I can pop in a line to business rules if you’re happy that it’s non-normative.

  184. Ge0rG

    This is exactly the sort of thing I'd like discussed among server and client developers.

  185. Kev

    <p>A server MAY choose not to deliver offline messages to a client that has already queried their MAM archive and received the archived copies of those messages that would otherwise be delivered - while not required of an implementation, this is helpful to avoid duplicate messages for clients, so is suggested.</p> Seems safe enough, to me.

  186. Ge0rG

    Kev: great wording, +1

  187. Kev

    Also pushed.

  188. Kev

    I’ll raise the PR, then.

  189. Ge0rG

    Kev: please do!

  190. Kev

    I think that’s done now - https://github.com/xsf/xeps/pull/1104

  191. Kev

    I guess I *could* hit merge myself, but I’d probably better leave it for jonas’ in case I messed something up in the PR.

  192. jonas’

    yes please don't just hit merge

  193. jonas’

    we don't have any automation there and it'll confuse the heck out of me

  194. Ellenor Malik


  195. Ge0rG

    So I wanted to have a perma-link to "this year's compliance suite XEP" and it's not possible :(

  196. wurstsalat

    Ge0rG, nginx redirect?

  197. Ge0rG

    wurstsalat: something like https://xmpp.org/compliancesuite/ maybe?

  198. Ge0rG

    wurstsalat: who can create such a redirect?

  199. wurstsalat

    Ge0rG, this file (I think) https://github.com/xsf/xmpp.org/blob/master/deploy/xsf.conf

  200. Ge0rG

    wurstsalat: I'm not sure I should dare doing that without asking the A-Team, sorry, the i-team first.

  201. wurstsalat

    Ge0rG, sure! those redirects work when building the page via docker

  202. Ge0rG

    https://github.com/xsf/xmpp.org/pull/960 is for the i-team

  203. emus

    please dont merge untill we have the new website setup

  204. Ge0rG

    emus: better comment that on the PR

  205. emus


  206. phryk

    MattJ, Zash: Do you guys have a list which clients support the technically non-standard invite-based registrations you implemented? (i.e. prosody mod_invites)

  207. phryk

    making some progress with dynamically built setup guides for my services' site and need that info to recommend people how to do things depending on what OS they're on and what client i'm recommending per OS. :)

  208. MattJ

    phryk, I recently learned that it was 99% standardized as https://xmpp.org/extensions/xep-0445.html

  209. MattJ

    but it had somehow passed me by

  210. phryk

    If I recall the situation right you and Zash implemented that but it wasn't complete, so you implemented the necessary extra bits and added them to the XEP but the original author reverted those changes to the XEP.

  211. phryk

    also none of the known client doaps report support for it :P