jdev - 2022-12-27

  1. MattJ

    nicoco: the additional fields were added in a later version of the XEP. To be compliant with this version, an implementation MUST support those fields, and MUST advertise #extended.

  2. MattJ

    It's not about shaming anyone or not 🙂

  3. Alastair Hogge

    Well said.

  4. nicoco

    MattJ: thanks for the explanation. so if an implementation only supports 3, it urn:xmpp:mam:1 then, I get it. (the shaming part was a joke, no worries ;))

  5. nicoco

    MattJ: thanks for the explanation. so if an implementation only supports 3, it should advertise urn:xmpp:mam:1 then, I get it. (the shaming part was a joke, no worries ;))

  6. MattJ

    No, mam:2 did not define the extra 3 fields

  7. MattJ

    There are many mam:2 implementations in the wild that don't support the latest revision of the XEP

  8. pep.

    Is there a list of o payloads that shouldn't be broadcasted by a MUC? (to prevent impersonating it)

  9. jonas’


  10. pep.

    All the <{muc*}x/> stuff? And things with stamps? (which are?)

  11. jonas’

    anything in the MUC namespace obviously.

  12. jonas’

    and delay and stanza-id, yes

  13. pep.


  14. jonas’

    only if you advertise the occupant-id feature

  15. jonas’

    (likewise for stanza-id, btw)

  16. jonas’

    and delay and stanza-id must only be stripped if they attempt to impersonate your domain

  17. jonas’

    though you might want to strip them if they otherwise deanonymise the user

  18. jonas’

    oh that's fun

  19. jonas’

    I should not have said that out loud

  20. pep.


  21. pep.

    k. Well I guess it doesn't hurt removing them anyway if they're making look like I did it, even though clients should check disco

  22. pep.

    Ah that means I need to have access to state in this method.. to know my own domain :x

  23. pep.

    https://xmpp.org/extensions/xep-0045.html#security doesn't seem to mention this. /me adds it to the TODO

  24. jonas’

    what exactly?

  25. pep.

    A list of stuff not to broadcast.

  26. jonas’

    ah, stripping <{muc}x/>

  27. pep.

    https://code.bouah.net/pep/hanabi-muc/commit/ce640864cae80f4456ac67c0914e745a8215f658 that's about all I can do without knowledge of my own domain I guess :x

  28. pep.

    Next step.

  29. jonas’

    I see an "and" in the commit message :<

  30. jonas’

    (it's disguised as ;, but still)

  31. pep.


  32. pep.

    You have seen nothing.

  33. pep.

    https://xmpp.org/extensions/xep-0203.html#security < This mentions stripping fwiw

  34. pep.

    https://xmpp.org/extensions/xep-0359.html < This only says validating entities should check whether @from actually supports it.

  35. nicoco

    MattJ: I don't grasp it all tbh, but I'll just respect the MUST I guess ;)

  36. MattJ

    nicoco: I think you've got it, apart from that you implicitly made an assumption that the previous version of the XEP used mam:1

  37. MattJ

    We've had mam:1, mam:2, and mam:2+mam:2#extended

  38. pep.

    (mam:3! :p)

  39. pep.

    Just that it didn't want to be named mam:3

  40. MattJ

    The new changes were all additions, so switching the whole lot to mam:3 would have needlessly broken every existing implementation

  41. pep.

    I think it's exactly the same really. I hope implementations don't check 'urn:xmpp:mam:2' but '^uxn:xmpp:mam:2$'. So in any case they'd have multiple checks to do (or to change this regex)

  42. pep.

    (if they even use regex)

  43. MattJ

    I hope you're joking 🙂

  44. pep.

    What do you mean

  45. MattJ

    Maybe I misunderstood your statement

  46. jonas’


  47. MattJ

    I'm not sure what regex has to do with anything

  48. MattJ

    It has zero applicability here

  49. pep.

    regex or whatever. I hope they check for the full string to equal the namespace they're looking for

  50. pep.

    Not some partial equality

  51. MattJ

    They shouldn't be doing regex, no

  52. pep.

    That's not the point

  53. nicoco

    we all know what happens when you try to parse XML with regular expressions, don't we?

  54. nicoco

    we all know what happens when we try to parse XML with regular expressions, don't we?

  55. pep.

    Forget I even said anything about regex, it doesn't matter to make this point

  56. MattJ

    I'm happy to say I've never used regex to parse XML, and I would never do so

  57. MattJ

    The sender retracted their previous message but your client does not support that

  58. jonas’

    I cannot say the same.

  59. MattJ

    I did it yesterday, with sed, in a bash script (it was a config file though, not XMPP)

  60. pep.

    So I stand by my comment above, I see no different in using mam:2#extended or mam:3. It's exactly the same change to me for a client to support it

  61. pep.

    (or ignore it)

  62. pep.


  63. Holger

    MattJ: I still believe that MUST can be slightly confusing as it may mislead the reader to believe the client can rely on those fields. Yes if you strictly follow the text then you'll probably get that you should check for the :2#extended feature. But if I see a MUST I'll always assume it was added to make a protocol/interop guarantee, which isn't the case here. So if I knew nothing about 0313 history it'd confuse me as well.

  64. Holger

    No big thing of course.

  65. MattJ

    pep. [09:17]: > So I stand by my comment above, I see no different in using mam:2#extended or mam:3. It's exactly the same change to me for a client to support it The difference isn't for clients that are updated to support it, the difference is for clients that are not updated - they continue to work

  66. pep.

    MattJ, they would anyway, since "mam:2" definitely isn't the same string as "mam:2#extended"

  67. MattJ

    Ah, so you're saying the XEP update should have said you MUST advertise mam:2 and mam:3

  68. MattJ

    Could have done that, yes

  69. pep.

    Ah, yeah

  70. pep.

    Ok sorry if that was the confusion

  71. MattJ

    That would probably have been confusing, as people might have tried using mam:3 in stanzas

  72. MattJ

    #extended is just a disco feature

  73. MattJ

    The core namespace remained mam:2

  74. pep.

    It's just names and conventions. You make them do whatever you want to :)

  75. MattJ

    And we did 🙂

  76. pep.

    I'm not saying it doesn't make sense, it's fine this way. Just that there wasn't much more differences to me to use mam2/3

  77. MattJ

    But it wasn't done only to make upgrading easy, it was done to keep compatibility with existing clients (given that 98% of the protocol is the same)

  78. pep.

    It looks like you're saying mam2/3 wouldn't keep compatibility and I'm saying that's not true. In both cases a server zould advertise both features right?

  79. pep.

    (if it supported both)

  80. MattJ

    It's not as simple as that, unfortunately

  81. pep.

    Ok I don't understand then

  82. pep.

    But it's fine, we can drop it if it gets too long :P

  83. MattJ

    We had implementation experience of supporting multiple namespaces concurrently with e.g. XEP-0198

  84. MattJ

    It's certainly possible, and perfectly fine if it's necessary. But it's not as trivial as it seems on the surface, and there was no need to do it in this case.

  85. pep.

    I actually don't get how it differs technically. 'mam:2#extended' is another version, same as 'mam:3'

  86. MattJ

    No, note how in the XEP, all the elements are still in the mam:2 namespace

  87. pep.

    Ok, it just makes slightly more sense to name it mam:2#foo, I guess, but that doesn't change that you could have advertised anything to say "we support this" and still used mam:2 in the spec

  88. pep.

    It's just a flag, the name doesn't really matter. It makes it easier for humans for sure (even though some may wonder why you're not using #extended then as the ns :p)

  89. MattJ

    Yes, you're totally right

  90. MattJ

    Given the choice though, I think #extended was a better fit because advertising mam:3 would lead people to assume that's the namespace of the elements too. They might assume that with #extended too of course, but less likely I'd hope.

  91. MattJ

    All we'd need was one server accidentally supporting mam:3 elements and then we'd soon have one or more clients doing it too 🙂

  92. pep.

    Holger, that "if .. MUST" doesn't disturb me that much. As an analogy I think I prefer !(foo && bar) rather than !foo || !bar. Plus in a spec it can quickly get messy to know what goes with what :/

  93. Holger

    Not sure we mean the same thing? My point is, for me a MUST implies a guarantee, i.e. protocol breakage if it's ignored. In this case there's no breakage if it's ignored, the guarantee only depends on the XEP revision, which is not a protocol thing. So the MUST may be confusing while trying to understand the protocol.

  94. pep.

    Well the MUST is behind a predicate here

  95. jonas’

    Holger, no it doesn't depend on the XEP revisino, it depends on a feature.

  96. Holger

    jonas’: The text says "servers MUST support (and announce) the feature".

  97. pep.

    The text says "Servers supporting"

  98. Holger

    It doesn't say "if the feature is announced, it MUST be supported".

  99. pep.

    It could be made slightly more explicit maybe, with an if

  100. pep.

    Maybe we should stick to simple english in specs, to make that less confusing and more accessible :/

  101. jonas’

    Holger, > Servers supporting fields marked with an asterisk (*) MUST advertise the disco feature 'urn:xmpp:mam:2#extended' and clients that depend on these fields MUST verify that the server advertises this feature before attempting to use them.

  102. pep.

    (That'd definitely help me too)

  103. Holger

    > Six further fields are defined by this XEP and MUST be supported by servers, though all of them are optional for the client.

  104. Holger

    nicoco pointed that out yesterday. I didn't remember it. Which is probably also due to the fact that it makes no sense to me 🙂

  105. jonas’

    Holger, https://xmpp.org/extensions/attic/xep-0313-0.6.html#filter

  106. jonas’

    the non-asterisk fields were always part of MAM

  107. Holger

    Sure, but those are only three?

  108. jonas’


  109. jonas’


  110. jonas’

    that does indeed sound like a weird thing then

  111. jonas’

    MattJ, wording issue or intentional? :)

  112. jonas’

    ("Six fields MUST be supported; servers supporting the last three MUST announce feature X". I would've expected: "Three fields MUST be supported, the other three fields MUST be supported iff the feature is announced")

  113. Holger

    đź‘Ť If the real intention simply was to encourage servers to offer them, it could just be stated that way, in my book.

  114. MattJ

    It's not encouragement, they are mandatory

  115. MattJ

    As is #extended

  116. pep.

    It's there because.. some previous version of mam:2 didn't have them?

  117. jonas’

    MattJ, but that's a weird thing to say if "clients that depend on these fields MUST verify that the server advertises this feature before attempting to use them"

  118. jonas’

    but I get your perspective now

  119. jonas’

    with mam:2 as one version, and mam:2+mam:2#extended as the next version.

  120. jonas’

    interesting way to look at things, I shall keep that in mind.

  121. pep.

    It's there because.. some previous version of 313 didn't have them?

  122. MattJ

    The alternative was mam:3 and breaking the world, or making these fields perpetually optional

  123. Holger

    Effectively they _are_ optional until we bump to mam:3.

  124. MattJ

    How so?

  125. MattJ

    It's a federated open ecosystem. By that logic, everything ever is always optional 🙂

  126. pep.

    And even the bump to mam:3 would be optional :P

  127. MattJ

    This is no different from bumping to mam:3 really, as pep said... they are just strings

  128. Holger

    Adhering to a MUST is not "optional".

  129. Holger

    In my book.

  130. MattJ

    The only difference is backwards compatibility is gained for legacy implementations

  131. MattJ

    Agreed, but you just wrote that it is

  132. Holger

    Well I expressed my concerns above, guess I won't bring them over better by adding even more text 🙂 Call it a day.

  133. pep.

    Holger, it is. It depends on time, motivation, money, [insert something here]. And while that's not implemented you're just "not compliant"(tm), or half-compliant

  134. pep.

    (or something-compliant)

  135. MattJ

    A server cannot be compliant with the latest XEP version without supporting those things

  136. MattJ

    They are not optional

  137. MattJ

    However implementations may not be up to date with the latest version of the XEP (whether it uses mam:3 or #extended)

  138. Holger

    Yes I get that idea. I just think it's confusing to do offer a guarantee that depends on a XEP revision, as the latter isn't a protocol thing.

  139. MattJ

    We do that all the time though

  140. MattJ

    I think you're only confused because we didn't call it mam:3

  141. Holger

    We may disagree but that's definitely not my confusion, no.

  142. MattJ


  143. MattJ

    So a XEP can never mandate stuff in a new revision?

  144. pep.

    Or from which point of its lifecycle (knowing that MAM was experimental for a really long time while being implemented in many places)

  145. Holger

    MattJ: If ignoring the new MUST breaks nothing then my answer would be no.

  146. pep.

    no it can't mandate stuff in newer revisions?

  147. MattJ

    I think that's where we disagree then

  148. Holger

    Not without actually bumping the namespace, no.

  149. pep.

    Well it has been bumped here

  150. MattJ

    This happens widely in protocols

  151. MattJ

    Like how TLS 1.3 has many new MUSTs, but still advertises as TLS 1.2 in the handshake for compatibility purposes

  152. MattJ

    In practice many things still accept 1.2 right now, but that won't always be the case

  153. flow

    doesn't tls 1.3 still negotiate that it's 1.3 despite the fact that the TLS version announced in the hello message is still 1.2?

  154. kapad

    as an somehow 'outsider', means that dont deal in developing xmpp server/clients, but also have read some of the xeps around, i dare to say that i feel a xep never give the choice how you gonna advertise a feature, but how you must. xep describes both the implementation and the language between xmpp entities. and because that lang spoken by machines must follow syntax at any cost

  155. flow

    > MattJ> So a XEP can never mandate stuff in a new revision? It is less about the revision, but about mandating new stuff without a namespace bump. And even there, it depends on the "new stuff"

  156. flow

    IMHO the mam xep could have simply strongly encouraged mam:2#extended, instead of mandating it with a new revision under the same namespace

  157. pep.

    There is a namespace bump

  158. flow

    I obviously don't see it so feel free to enlighten me, it appears the namespace is still mam:2

  159. pep.

    The bump is #extended

  160. flow

    that's one way to view it, but certainly not the tranditional namespace bump kind

  161. pep.

    As I said earlier, it's as if the spec had said "advertize both mam:3 and mam:2 to stay compatible"

  162. flow

    pep., fwiw, I agree with MattJ that bumping to mam:3 just because there fields where added whould have been destructive

  163. flow

    because then suddenly all mam:2-only implementations could no longer use mam:3-only services, for no good reason

  164. pep.

    I think you've missed my point then, but it's fine

  165. flow

    if you "extend" a spec, by basically optional fields, then the mam:2#extended approach is sensible

  166. flow

    pep., very well possible, it was a large backlog to read ;)

  167. pep.

    You can "extend" a spec naming the extended ns whatever you like, here it's "mam:2#extended", it could have been "mam:3"

  168. pep.

    In both cases the spec says you advertize "mam:2" as well to stay compatible, and you're done

  169. pep.

    Anyway, there is a bump to me here

  170. pep.

    Just not the kind we're used to see (numbers) but it's the exact same thing to me. The string changes

  171. pep.

    Just that 313 defines "Compatibility with the spec" as "You have implemented all-the-things". I doubt generally we see this wording in other specs? But I don't really see an issue with it.

  172. pep.

    It "only" prevents you to call your impl fully compatible if it doesn't implement everything

  173. jonas’

    I don't think that sticking to the notion of "one namespace per document version" is a good thing

  174. jonas’

    it's counter to the extensibility idea

  175. pep.

    I agree

  176. jonas’

    as long as elements don't change incompatibly, we should stick with the same identifier

  177. jonas’

    (this has been done in MIX IIRC)

  178. jonas’

    so this step, with adding mam:2#extended *and* mandating it in the current XEP version (which was, by the way, before stabilization as 1.0), is very sensible to me

  179. jonas’

    so this step, with adding mam:2#extended *and* mandating it in the current XEP version (which was, by the way, before stabilization as 1.0) without changing anything existing as mam:2, is very sensible to me

  180. jonas’

    and using mam:2#extended instead of mam:3 is also good because it clearly shows that this is not the traditional way we've handled namespace versioning

  181. jonas’

    so all fine by me

  182. flow

    mandating #extended in the current MAM version gains you nothing compared to just strongly encouraging it, instead it just causes confusion.

  183. flow

    simply because there is no gurantee that there isn't an implementation out there that never has seen the new XEP revision and just announces mam:2

  184. jonas’

    flow, I think quite the contrary

  185. jonas’

    to someone coming newly to the spec, it's better to mandate it.

  186. jonas’

    that's ok, clients are clearly told they need to check for #extended

  187. flow

    why? the spec could simply say "if you are working on an mam implementation, then you really should also implement #extended"

  188. flow

    that would have the same effect

  189. jonas’

    that are more words than "MUST implement this"

  190. jonas’

    less words good

  191. jonas’

    you'd still need the notice that client devs need to check for the feature explicitly

  192. flow

    instead, what we have causes confusion, as can be seen by the question that started this discussion

  193. Holger

    "Why on earth would I need to check for #extended if it's mandatory?"

  194. flow

    what Holger says

  195. pep.

    Because specs are living documents :x

  196. pep.

    Things change

  197. flow

    pep., that shouldn't be an excuse that they can be confusing

  198. flow

    or even cause interoperability issues

  199. jonas’

    we can add "because it was added as an extension to mam:2"

  200. jonas’

    but I'm not sure that helps anyone

  201. MattJ

    Holger [11:21]: > "Why on earth would I need to check for #extended if it's mandatory?" Why check for mam:3 if it's mandatory?

  202. MattJ

    It's the same difference

  203. pep.

    s/ for mam:3/ even :P

  204. flow

    I don't see how mam:3 compares to the mam:2 + #extended situation

  205. jonas’

    imagine those form fields had been a separate document

  206. MattJ

    I don't see how it differs

  207. jonas’

    and it'll become clear

  208. flow

    mam:2 can work without #extended, it is a prime example of an extension that does not break backwards compatibility on the protcol layer

  209. MattJ

    mam:1 still works if implemented, too

  210. Holger

    MattJ: Are you talking mam:3 instead of #extended or mam:3 instead of mam:2+#extended?

  211. MattJ

    Either works

  212. flow

    mam:3 instead of #extended would be pretty strange and not idomatic

  213. Holger

    MattJ: The former breaks compat, obviously.

  214. MattJ

    A client that depends on the stuff in the latest revision can check only for #extended

  215. flow

    and mam:3 instead of mam:2+#extended would be destructive, as in, cause interoperability issues for no reason

  216. Holger

    MattJ: Yes. I'm not arguing that something doesn't work. Just that it's slightly confusing when trying to grasp the spec.

  217. jonas’

    Holger, I guess PRs welcome then

  218. pep.

    "jonas’> imagine those form fields had been a separate document" < this really

  219. flow

    I am not sure what would change if those form fields would be specified separate document?

  220. Holger

    Would 0313 mandate to implement it?

  221. flow

    but I view them as something that could have been specified in an extra document (like the various pubsub extensions)

  222. flow

    not sure how this is relevant for the discussion, though

  223. jonas’

    flow, if they had been a separate document, nobody would be confused about the "mandatory fields and check for this feature and don't rely on it unless you see that feature" thing

  224. flow

    if they where in a separte document, then it would also be strange if the mam xep says that you also need to implement the mam:2#extended xep…

  225. jonas’

    right, it wouldn't say that

  226. jonas’

    instead we'd have an indiscoverable mess

  227. flow

    why would it be indiscoverable?

  228. jonas’

    how would you learn that XEP-5555 is relevant to you, when just looking at XEP-0313?

  229. flow

    forward pointer in the mam xep?

  230. MattJ

    Because MAM would no longer mean "XEP-0313"

  231. flow

    similar to pubsub does not just mean xep60

  232. jonas’

    which isn't exactly great

  233. MattJ

    Once you add the pointer and say it's mandatory, might as well put it in the same doc 🙂

  234. flow

    right, but I am arguing that saying it is mandatory gains you nothing and only causes confusion, compared to just strongly recommending it

  235. pep.

    (Also how MUC doesn't just mean 0045. /me looks at https://wiki.xmpp.org/web/MUC_Extensions )

  236. pep.

    "Once you add the pointer and say it's mandatory, might as well put it in the same doc" < "It depends"(tm)

  237. pep.

    The doc gets huge and messy, you might want to split it up

  238. pep.

    Just that the xep discoverability story is meh

  239. flow

    right, but I am arguing that saying mam:2#extended is mandatory gains you nothing and only causes confusion, compared to just strongly recommending mam:2#extended

  240. Zash

    flow, if you don't implement #extended then what you have is https://xmpp.org/extensions/attic/xep-0313-0.6.3.html

  241. Holger

    I (we?) are arguing this MUST to be confusing, at least to newcomers. There's an 'extended' feature that servers MUST implement and announce. To me, the obvious question would be "why do clients need to check for it if it's mandatory?" Even if I'm on the right track and assume spec evolution to be the culprit, I'd go on "then why is it mandatory now, am I missing something?". I may be wrong about this potential confusion. But I'm confused how you guys keep insisting on explaining the status quo, rather than explaining why you see no potential for confusion 🙂

  242. Holger

    I mean this entire topic came up due to the confusion I have in mind.

  243. jonas’

    propose wording changes

  244. jonas’

    best way to discuss stuff is when looking at a diff

  245. Holger

    Yeah, except that I'm not sure how to word the reason for it being mandatory now.

  246. Holger

    But I can propose wording that explains how the additional feature exists due to spec evolution, yes.

  247. MattJ

    > rather than explaining why you see no potential for confusion I can't say I don't see potential for confusion, rather I see potential for confusion everywhere I look for it. I'm happy to find ways to reduce confusion, but you have to draw a line somewhere

  248. MattJ

    Adding more words about a couple of simple MUSTs also has potential to cause confusion