jdev - 2022-12-27

nicoco: the additional fields were added in a later version of the XEP. To be compliant with this version, an implementation MUST support those fields, and MUST advertise #extended.

It's not about shaming anyone or not 🙂

Well said.

MattJ: thanks for the explanation. so if an implementation only supports 3, it should advertise urn:xmpp:mam:1 then, I get it. (the shaming part was a joke, no worries ;)) ✏

No, mam:2 did not define the extra 3 fields

There are many mam:2 implementations in the wild that don't support the latest revision of the XEP

Is there a list of o payloads that shouldn't be broadcasted by a MUC? (to prevent impersonating it)

no

All the <{muc*}x/> stuff? And things with stamps? (which are?)

anything in the MUC namespace obviously.

and delay and stanza-id, yes

occupant-id

only if you advertise the occupant-id feature

(likewise for stanza-id, btw)

and delay and stanza-id must only be stripped if they attempt to impersonate your domain

though you might want to strip them if they otherwise deanonymise the user

oh that's fun

I should not have said that out loud

heh

k. Well I guess it doesn't hurt removing them anyway if they're making look like I did it, even though clients should check disco

Ah that means I need to have access to state in this method.. to know my own domain :x

https://xmpp.org/extensions/xep-0045.html#security doesn't seem to mention this. /me adds it to the TODO

what exactly?

A list of stuff not to broadcast.

ah, stripping <{muc}x/>

https://code.bouah.net/pep/hanabi-muc/commit/ce640864cae80f4456ac67c0914e745a8215f658 that's about all I can do without knowledge of my own domain I guess :x

Next step.

I see an "and" in the commit message :<

(it's disguised as ;, but still)

ssshhh

You have seen nothing.

https://xmpp.org/extensions/xep-0203.html#security < This mentions stripping fwiw

https://xmpp.org/extensions/xep-0359.html < This only says validating entities should check whether @from actually supports it.

MattJ: I don't grasp it all tbh, but I'll just respect the MUST I guess ;)

nicoco: I think you've got it, apart from that you implicitly made an assumption that the previous version of the XEP used mam:1

We've had mam:1, mam:2, and mam:2+mam:2#extended

(mam:3! :p)

Just that it didn't want to be named mam:3

The new changes were all additions, so switching the whole lot to mam:3 would have needlessly broken every existing implementation

I think it's exactly the same really. I hope implementations don't check 'urn:xmpp:mam:2' but '^uxn:xmpp:mam:2$'. So in any case they'd have multiple checks to do (or to change this regex)

(if they even use regex)

I hope you're joking 🙂

What do you mean

Maybe I misunderstood your statement

excusemewhat

I'm not sure what regex has to do with anything

It has zero applicability here

regex or whatever. I hope they check for the full string to equal the namespace they're looking for

Not some partial equality

They shouldn't be doing regex, no

That's not the point

we all know what happens when we try to parse XML with regular expressions, don't we? ✏

Forget I even said anything about regex, it doesn't matter to make this point

I'm happy to say I've never used regex to parse XML, and I would never do so

The sender retracted their previous message but your client does not support that

I cannot say the same.

I did it yesterday, with sed, in a bash script (it was a config file though, not XMPP)

So I stand by my comment above, I see no different in using mam:2#extended or mam:3. It's exactly the same change to me for a client to support it

(or ignore it)

difference*

MattJ: I still believe that MUST can be slightly confusing as it may mislead the reader to believe the client can rely on those fields. Yes if you strictly follow the text then you'll probably get that you should check for the :2#extended feature. But if I see a MUST I'll always assume it was added to make a protocol/interop guarantee, which isn't the case here. So if I knew nothing about 0313 history it'd confuse me as well.

No big thing of course.

pep. [09:17]: > So I stand by my comment above, I see no different in using mam:2#extended or mam:3. It's exactly the same change to me for a client to support it The difference isn't for clients that are updated to support it, the difference is for clients that are not updated - they continue to work

MattJ, they would anyway, since "mam:2" definitely isn't the same string as "mam:2#extended"

Ah, so you're saying the XEP update should have said you MUST advertise mam:2 and mam:3

Could have done that, yes

Ah, yeah

Ok sorry if that was the confusion

That would probably have been confusing, as people might have tried using mam:3 in stanzas

#extended is just a disco feature

The core namespace remained mam:2

It's just names and conventions. You make them do whatever you want to :)

And we did 🙂

I'm not saying it doesn't make sense, it's fine this way. Just that there wasn't much more differences to me to use mam2/3

But it wasn't done only to make upgrading easy, it was done to keep compatibility with existing clients (given that 98% of the protocol is the same)

It looks like you're saying mam2/3 wouldn't keep compatibility and I'm saying that's not true. In both cases a server zould advertise both features right?

(if it supported both)

It's not as simple as that, unfortunately

Ok I don't understand then

But it's fine, we can drop it if it gets too long :P

We had implementation experience of supporting multiple namespaces concurrently with e.g. XEP-0198

It's certainly possible, and perfectly fine if it's necessary. But it's not as trivial as it seems on the surface, and there was no need to do it in this case.

I actually don't get how it differs technically. 'mam:2#extended' is another version, same as 'mam:3'

No, note how in the XEP, all the elements are still in the mam:2 namespace

Ok, it just makes slightly more sense to name it mam:2#foo, I guess, but that doesn't change that you could have advertised anything to say "we support this" and still used mam:2 in the spec

It's just a flag, the name doesn't really matter. It makes it easier for humans for sure (even though some may wonder why you're not using #extended then as the ns :p)

Yes, you're totally right

Given the choice though, I think #extended was a better fit because advertising mam:3 would lead people to assume that's the namespace of the elements too. They might assume that with #extended too of course, but less likely I'd hope.

All we'd need was one server accidentally supporting mam:3 elements and then we'd soon have one or more clients doing it too 🙂

Holger, that "if .. MUST" doesn't disturb me that much. As an analogy I think I prefer !(foo && bar) rather than !foo || !bar. Plus in a spec it can quickly get messy to know what goes with what :/

Not sure we mean the same thing? My point is, for me a MUST implies a guarantee, i.e. protocol breakage if it's ignored. In this case there's no breakage if it's ignored, the guarantee only depends on the XEP revision, which is not a protocol thing. So the MUST may be confusing while trying to understand the protocol.

Well the MUST is behind a predicate here

Holger, no it doesn't depend on the XEP revisino, it depends on a feature.

jonas’: The text says "servers MUST support (and announce) the feature".

The text says "Servers supporting"

It doesn't say "if the feature is announced, it MUST be supported".

It could be made slightly more explicit maybe, with an if

Maybe we should stick to simple english in specs, to make that less confusing and more accessible :/

Holger, > Servers supporting fields marked with an asterisk (*) MUST advertise the disco feature 'urn:xmpp:mam:2#extended' and clients that depend on these fields MUST verify that the server advertises this feature before attempting to use them.

(That'd definitely help me too)

> Six further fields are defined by this XEP and MUST be supported by servers, though all of them are optional for the client.

nicoco pointed that out yesterday. I didn't remember it. Which is probably also due to the fact that it makes no sense to me 🙂

Holger, https://xmpp.org/extensions/attic/xep-0313-0.6.html#filter

the non-asterisk fields were always part of MAM

Sure, but those are only three?

yes

ohhh

that does indeed sound like a weird thing then

MattJ, wording issue or intentional? :)

("Six fields MUST be supported; servers supporting the last three MUST announce feature X". I would've expected: "Three fields MUST be supported, the other three fields MUST be supported iff the feature is announced")

👍 If the real intention simply was to encourage servers to offer them, it could just be stated that way, in my book.

It's not encouragement, they are mandatory

As is #extended

MattJ, but that's a weird thing to say if "clients that depend on these fields MUST verify that the server advertises this feature before attempting to use them"

but I get your perspective now

with mam:2 as one version, and mam:2+mam:2#extended as the next version.

interesting way to look at things, I shall keep that in mind.

It's there because.. some previous version of 313 didn't have them? ✏

The alternative was mam:3 and breaking the world, or making these fields perpetually optional

Effectively they _are_ optional until we bump to mam:3.

How so?

It's a federated open ecosystem. By that logic, everything ever is always optional 🙂

And even the bump to mam:3 would be optional :P

This is no different from bumping to mam:3 really, as pep said... they are just strings

Adhering to a MUST is not "optional".

In my book.

The only difference is backwards compatibility is gained for legacy implementations

Agreed, but you just wrote that it is

Well I expressed my concerns above, guess I won't bring them over better by adding even more text 🙂 Call it a day.

Holger, it is. It depends on time, motivation, money, [insert something here]. And while that's not implemented you're just "not compliant"(tm), or half-compliant

(or something-compliant)

A server cannot be compliant with the latest XEP version without supporting those things

They are not optional

However implementations may not be up to date with the latest version of the XEP (whether it uses mam:3 or #extended)

Yes I get that idea. I just think it's confusing to do offer a guarantee that depends on a XEP revision, as the latter isn't a protocol thing.

We do that all the time though

I think you're only confused because we didn't call it mam:3

We may disagree but that's definitely not my confusion, no.

Ok

So a XEP can never mandate stuff in a new revision?

Or from which point of its lifecycle (knowing that MAM was experimental for a really long time while being implemented in many places)

MattJ: If ignoring the new MUST breaks nothing then my answer would be no.

no it can't mandate stuff in newer revisions?

I think that's where we disagree then

Not without actually bumping the namespace, no.

Well it has been bumped here

This happens widely in protocols

Like how TLS 1.3 has many new MUSTs, but still advertises as TLS 1.2 in the handshake for compatibility purposes

In practice many things still accept 1.2 right now, but that won't always be the case

doesn't tls 1.3 still negotiate that it's 1.3 despite the fact that the TLS version announced in the hello message is still 1.2?

as an somehow 'outsider', means that dont deal in developing xmpp server/clients, but also have read some of the xeps around, i dare to say that i feel a xep never give the choice how you gonna advertise a feature, but how you must. xep describes both the implementation and the language between xmpp entities. and because that lang spoken by machines must follow syntax at any cost

> MattJ> So a XEP can never mandate stuff in a new revision? It is less about the revision, but about mandating new stuff without a namespace bump. And even there, it depends on the "new stuff"

IMHO the mam xep could have simply strongly encouraged mam:2#extended, instead of mandating it with a new revision under the same namespace

There is a namespace bump

I obviously don't see it so feel free to enlighten me, it appears the namespace is still mam:2

The bump is #extended

that's one way to view it, but certainly not the tranditional namespace bump kind

As I said earlier, it's as if the spec had said "advertize both mam:3 and mam:2 to stay compatible"

pep., fwiw, I agree with MattJ that bumping to mam:3 just because there fields where added whould have been destructive

because then suddenly all mam:2-only implementations could no longer use mam:3-only services, for no good reason

I think you've missed my point then, but it's fine

if you "extend" a spec, by basically optional fields, then the mam:2#extended approach is sensible

pep., very well possible, it was a large backlog to read ;)

You can "extend" a spec naming the extended ns whatever you like, here it's "mam:2#extended", it could have been "mam:3"

In both cases the spec says you advertize "mam:2" as well to stay compatible, and you're done

Anyway, there is a bump to me here

Just not the kind we're used to see (numbers) but it's the exact same thing to me. The string changes

Just that 313 defines "Compatibility with the spec" as "You have implemented all-the-things". I doubt generally we see this wording in other specs? But I don't really see an issue with it.

It "only" prevents you to call your impl fully compatible if it doesn't implement everything

I don't think that sticking to the notion of "one namespace per document version" is a good thing

it's counter to the extensibility idea

I agree

as long as elements don't change incompatibly, we should stick with the same identifier

(this has been done in MIX IIRC)

so this step, with adding mam:2#extended *and* mandating it in the current XEP version (which was, by the way, before stabilization as 1.0) without changing anything existing as mam:2, is very sensible to me ✏

and using mam:2#extended instead of mam:3 is also good because it clearly shows that this is not the traditional way we've handled namespace versioning

so all fine by me

mandating #extended in the current MAM version gains you nothing compared to just strongly encouraging it, instead it just causes confusion.

simply because there is no gurantee that there isn't an implementation out there that never has seen the new XEP revision and just announces mam:2

flow, I think quite the contrary

to someone coming newly to the spec, it's better to mandate it.

that's ok, clients are clearly told they need to check for #extended

why? the spec could simply say "if you are working on an mam implementation, then you really should also implement #extended"

that would have the same effect

that are more words than "MUST implement this"

less words good

you'd still need the notice that client devs need to check for the feature explicitly

instead, what we have causes confusion, as can be seen by the question that started this discussion

"Why on earth would I need to check for #extended if it's mandatory?"

what Holger says

Because specs are living documents :x

Things change

pep., that shouldn't be an excuse that they can be confusing

or even cause interoperability issues

we can add "because it was added as an extension to mam:2"

but I'm not sure that helps anyone

Holger [11:21]: > "Why on earth would I need to check for #extended if it's mandatory?" Why check for mam:3 if it's mandatory?

It's the same difference

s/ for mam:3/ even :P

I don't see how mam:3 compares to the mam:2 + #extended situation

imagine those form fields had been a separate document

I don't see how it differs

and it'll become clear

mam:2 can work without #extended, it is a prime example of an extension that does not break backwards compatibility on the protcol layer

mam:1 still works if implemented, too

MattJ: Are you talking mam:3 instead of #extended or mam:3 instead of mam:2+#extended?

Either works

mam:3 instead of #extended would be pretty strange and not idomatic

MattJ: The former breaks compat, obviously.

A client that depends on the stuff in the latest revision can check only for #extended

and mam:3 instead of mam:2+#extended would be destructive, as in, cause interoperability issues for no reason

MattJ: Yes. I'm not arguing that something doesn't work. Just that it's slightly confusing when trying to grasp the spec.

Holger, I guess PRs welcome then

"jonas’> imagine those form fields had been a separate document" < this really

I am not sure what would change if those form fields would be specified separate document?

Would 0313 mandate to implement it?

but I view them as something that could have been specified in an extra document (like the various pubsub extensions)

not sure how this is relevant for the discussion, though

flow, if they had been a separate document, nobody would be confused about the "mandatory fields and check for this feature and don't rely on it unless you see that feature" thing

if they where in a separte document, then it would also be strange if the mam xep says that you also need to implement the mam:2#extended xep…

right, it wouldn't say that

instead we'd have an indiscoverable mess

why would it be indiscoverable?

how would you learn that XEP-5555 is relevant to you, when just looking at XEP-0313?

forward pointer in the mam xep?

Because MAM would no longer mean "XEP-0313"

similar to pubsub does not just mean xep60

which isn't exactly great

Once you add the pointer and say it's mandatory, might as well put it in the same doc 🙂

(Also how MUC doesn't just mean 0045. /me looks at https://wiki.xmpp.org/web/MUC_Extensions )

"Once you add the pointer and say it's mandatory, might as well put it in the same doc" < "It depends"(tm)

The doc gets huge and messy, you might want to split it up

Just that the xep discoverability story is meh

right, but I am arguing that saying mam:2#extended is mandatory gains you nothing and only causes confusion, compared to just strongly recommending mam:2#extended ✏

flow, if you don't implement #extended then what you have is https://xmpp.org/extensions/attic/xep-0313-0.6.3.html

I (we?) are arguing this MUST to be confusing, at least to newcomers. There's an 'extended' feature that servers MUST implement and announce. To me, the obvious question would be "why do clients need to check for it if it's mandatory?" Even if I'm on the right track and assume spec evolution to be the culprit, I'd go on "then why is it mandatory now, am I missing something?". I may be wrong about this potential confusion. But I'm confused how you guys keep insisting on explaining the status quo, rather than explaining why you see no potential for confusion 🙂

I mean this entire topic came up due to the confusion I have in mind.

propose wording changes

best way to discuss stuff is when looking at a diff

Yeah, except that I'm not sure how to word the reason for it being mandatory now.

But I can propose wording that explains how the additional feature exists due to spec evolution, yes.

> rather than explaining why you see no potential for confusion I can't say I don't see potential for confusion, rather I see potential for confusion everywhere I look for it. I'm happy to find ways to reduce confusion, but you have to draw a line somewhere

Adding more words about a couple of simple MUSTs also has potential to cause confusion