XSF Discussion - 2020-11-19

flow, > This PR introduces a breaking change of the wire protocol and hence requires a namespace bump. that's not stricly true for experimental XEPs though, right?

yes

i mean whether or not it's a good idea to bump can be up to debate. but there is no rule that says you have to

(and in that particular case I don’t think it's a good idea to bump)

What now?

https://github.com/xsf/xeps/pull/998

Daniel, every xep with a number should undergo a namespace bump if a breakking wire protocol change is introduced

flow, according to your opinion or according to some rule?

flow, quote?

Can a thing implementing the old version interop with something implementing the new version?

I think that is what was practiced in the past (look for example at the jingle namespace, I'd assume a large part was done while the xep was experimental)

Zash, in this case the answer is no

the fact that individual authors of MAM and jingle decided to bump doesn’t mean that's always a good idea

Daniel, I think it is always a good idea as soon as the xep becomes a number

speaking broadly here (looking a bit at MAM) I as a client author often find it easier to just look for a different element or look out for the subtle differences between protocol versions instead implementing a whole new namespace. that might be due to how my library is set up

but code for "is attribute x here if not behave slighlty different" is often less than replicating objects

One of our objectives is to "ensure interoperability", I think we would violate that objective if we did breaking changes to the wire protocol without a namespace bump in official XEPs

> whether or not it's a good idea to bump can be up to debate. It is never a good idea to bump.

anyhow in the particular case of 420+omemo; nobody implements the current version

so we end up increasing the namespace version to infinity without practical use

Daniel, that is an argument that an namespace bump does not incur a high cost in this case

flow, we should ensure interop for anything that is Draft+

(in which case we don't namespace bump, because namespace bump is not interoperable)

Daniel, I think we should ensure interop for every standard we adopted, but we should introduce a supervised place for ProtoXEPs where things can be changed without a nemsapce bump

that's called 'experimental'

I think we are talking more or less about the same thing: we need a place where XEPs can be easily modified without a namespace bump, but I think if experimental becomes that place, then it would eventually hurd our reputation

"look XMPP is heavily modular and now even the single modules don't interoperate"

experimental doesn’t become that place it is that place

has been for 20 years

flow: you remember that time when we tried to bump Carbons?

I'd guess that most developers that implement experimental XEPs are well aware that they might have breaking changes...

that's a guess. I for example wouldn't expect that

isn't there a bold warning above experimental XEPs, telling they are only for exploratory implementations and not production?

larma, there is, but that doesn't imply that after you determined via disco#info lookup that an remote entity supports the xep, it actually doesn't

with the number of Experimental XEPs running in production, people must be ignoring those warnings

I think that's a failure on our part that we haven’t been moving XEP ups the chain fast enough

again, yes I also want to have a curated place for ProtoXEPs which can undergo breaking changes without a namespace bump, but experimental isn't the place

a failure that we have slowly been trying to correct

flow: for most protocol changes a namespace bump is just the wrong tool. A new feature is much more elegant and well feasible for many protocol changes.

could be a simple as adding protoxeps/ to the xsf repo and render the XEPs within there

Ge0rG, if you can do it via a new feature then this is obviously the superior approach to a namespace bump

Ge0rG, but sadly I don't think this is possible in the PR in question

flow: I remember this same discussion from a year or two ago, and I'm pretty sure the conclusion was that all the tools are in place and we are just using Experimental and Draft wrong.

i'm not entirely sure that we reached a conclusion. but at least that wasn’t an unpopular opinion

well, there are two parties involved in "using XEP status", the author who might want to push a XEP forward, and the Council that ultimately makes the decision

we even introduced Document Shepherds who can push a XEP forward on behalf of the community.

but I don't see wide use of that yet.

Also community feedback to LCs and CFEs is... sparse

(yesterday it was discussed in council what happens when file-metadata needs a namespace bump. answer is: it doesn't. All elements are optional and adding new optional elements isn't backwards incompatible. Old implementations will just ignore the new elements. That's good enough for both experimental and draft status. In Final, new elements can be added via new XEPs by using a new namespace. So I'm not aiming to ever namespace bump that XEP.)

larma, yes, if I understand the case with file-metadata correctly, this wouldn't require a namespace bump: since all elements are optional, interop is easy, even if new optional elements are added this wouldn't require a namespace bump. only if new required elements are added

larma: what if you want to / have to rename elements?

Ge0rG, I guess it depends if interop would become an issue in that case

Beside that, we have clear rules on namespace bumps in an Active, Procedural XEP, so it's not like something for council to decide on. https://xmpp.org/extensions/xep-0053.html: > While the XEP is in the Experimental state, if appropriate and in consultation with the author(s), the XMPP Registrar shall update the namespace version number at the end of namespace (e.g., from "0" to "1"); the XMPP Registrar shall do so only if the protocol defined in the XEP has been modified in a way that is not backwards-compatible with an earlier version of the protocol.

Ge0rG, I hardly see how renaming would ever be needed...

larma: well, that was a rather hypothetical question indeed

> All elements are optional and adding new optional elements isn't backwards incompatible. That's not entirely true. Someone can write a validator that ensures that the payloads match the allowed protocol (and people do this), so adding new bits of allowed protocol is not backwards compatible with such implementations.

I have been banned from the kuketzblog muc without having written there anything. Does anyone else has the same problem?

Kev, I don't like that line of thinking. That would go against the "extensible" in XMPP

A validator can only validate what he knows, not what he doesn't know

And we shouldn't close the door on introducing new optional protocol parts without having to bump the namespace

flow: Extensible in XMPP has always meant in practice that you can add new namespaces, not that you can change what is allowable within a single namespace.

Kev, i would challenge that in two ways: first, i am not sure if this is true in the past, and secondly, I think it shouldn't be true in the present and future

Whether you like it or not, I'm asserting that this is how the specs are used. Not typically on Internet deployments, but very much in other deployments people care about such things :)

in any way, this should be documented

but I really don't like the idea what we can't even introduce new optional attributes without having a namespace bump

flow: We have had such discussions in the past, and I have 99% sure came to the conclusion I've described above, FWIW. And I do not disagree with documenting it.

We can still add new attributes, FWIW, but by negotiation.

We just can't send them unilaterally and expect that the result will be accepted.

Kev: but that effectively means that we can only add new attributes by version bump, not by new feature vars

because that would break strict parsers not knowing about the new feature

Ge0rG, I assume validators validate the structure and content kind, not the concrete content type

Kev, you'd say it needs to be specifically written in the XEP that more elements are allowed? Because if it is written, the validating entity is actually misbehaving by rejecting unknown elements ;)

"Future versions of this XEP may add new elements. Entities are therefor required to silently ignore all unknown elements."

larma: it'd be *very* interesting for EXI and other systems that require full knowledge of all supported elements

Ge0rG, how?

larma: Sorry, I was picking up the general point, not a particular XEP. If it says that, all is good.

It doesn't say it explicitly *yet*, but I'm open on adding it 😉

i feel this would probably lead to more and more future XEPs adding this a disclaimer, not sure if we shouldn't simply switch the default and have XEPs state that they won't add new things without a namespace bump

Is there some netiquette when to ban people?

flow, I agree on that at least for experimental XEPs. For Draft XEPs it might be not that appropriate: given that drafts include a schema, it should be possible to actually use that schema for validating/exi usage (although I personally think our current exi spec is overly complex and would be more likely to be actually implemented if severely simplified such that it's not doing any schema-based compression at all, but that's a different story)

larma, EXI can cope with incomplete schemas, that's not an issue. And XEP schemas describe that is there, not what is not there

actually I've seen many schemas describing what is not there...

larma, possibly, but given that we all aggree that one can plug in new elements under a different namespace (nearly) everywhere in xmpp…

larma, actually do you have an example for that?

https://xmpp.org/extensions/xep-0060.html item element

It has explicit "<xs:any namespace='##other'/>"

is that surprising given that <item/> is a container element for arbitrary payload elements?

No, but still it explicitly describes what should be default

I'm not convinced that moving to a world where you can't write validators because anything goes is an optimal direction. Maybe an Experimental/Draft split there would help, but equally that would work if you consider Experimental to not be deployed, and if you consider Experimental not to be deployed then namespace bumps are also not harmful.

But I don't have bandwidth to give it much thought this morning.

Should https://xmpp.org/extensions/xep-0353.html still be in LC?

2019

no.

I intend on '0001-restarting those expired LCs after the next elections

Kev, I think we have, in the past, assumed that adding an optional attribute in an existing namespace is OK. Whether we still should is another matter.

dwd: what about adding new elements in an existing namespace?

Ge0rG, I can't think of an example of that.

Ge0rG, Whereas, for adding an optional attribute, we did that in XEP-0237 (later folded into RFC 6121).

Is there a principal difference between adding elements and attributes, wrt strict schema validators?

Kev, FWIW, my understanding is that we have consistently said that if you use the schemas to validate traffic at runtime, you get to keep the pieces. As I say, we may want to change this, but it would be, I think, a change, rather than "moving to a world".

Ge0rG, I don't know of one. I think it might be marginally easier to deal with unexpected attributes when processing XML in code than unexpected elements, and we do assume that certain elements only contain a single child.

dwd: I'd say that adding unknown elements is a problem orthogonal to adding multiple elements where only one is allowed

I assume you are looking at the allowed content for IQ stanzas

Ge0rG, Probably, yes, but it might explain why we've avoided it, explicitly or not.

Ge0rG, I was indeed thinking that, and the question-mark over multiple elements within the XEP-0060 <item/> element.

One could argue that the semantic impact of unknown attributes is smaller than of unknown elements

Yes, and Kev notes that negotiation makes all this a lot safer.

Though I think XEP-0115 added attributes without negotiation.

Last year we had a very interesting what-if discussion in the context of steam compression and EXI, which I'd summarize as "a given client should exactly know the schema it can process, and could communicate that to the server on login, so that a custom minimal optimized exi or compression lookup table can be used for that connection, stripping all unknown elements / attributes at the server side "

Yup, I recall that. Lead mostly by arc I think.

Yup

That allows simple clients with hardwired EXI, such as microcontrollers

Simple client, complex server is a common xmpp protocol design principle

Ge0rG, But I don't think that a client requesting that unknown-to-it attributes are stripped by the server is breaking the end-to-end principle.

dwd: given that the client would ignore those attributes anyway, yes. But the same is true for elements

Ge0rG, Yes, it is. But I think we developed our unwritten rules of thumb around hand-parsing XML rather than anything else.

I agree with GeOrG, there's no purpose sending XML schema elements to clients which are not designed to understand them.

(And again: I'm not saying that our unwritten rules of thumb are right for today's developers)

arc, s/not designed/explicitly claim not/

Sure it's *nice* to have a xml console showing the raw traffic. That is an edge case that clients can ask for with EXI.

dwd: so what does that mean for writing down our unwritten rules of thumb?

Should Next Council create an Informational XEP about how and when to bump namespaces?

If one is submitted, Council will vote on it.

I seem to recall to have proposed to do that and it got shot down

because "it is obvious anyway"

despite such discussions coming up again and again

If it's obvious then it should be easy to write it down

jonas’: I don't remember your offer or it being shut down, but I think it's a good idea, especially if it covers schema validators and EXI

EXI literally has a flag to turn on/off whether you want strict schema or not. The problem of servers sending clients unexpected xml elements is one of bandwidth. The sender does not, and should not, know what bandwidth restrictions exist on the receiving side. For example, some IoT devices may turn off support for regular message body, ie chat. They might work only on pubsub and adhoc commands, and regular chat messages can cause their tiny battery powered cpu to wake up too often for them to run on tiny solar cell power.

arc: the same mechanism would help mobile clients. No need to send chat state notifications and wake up the device if the client can't display them anyway

So by stripping all unknown elements, then dropping empty containers, you'd get immediate optimization

Would have been nice if "supports normal chat messages" was in disco/caps

... 20 years ago

Ge0rG, tricky if the server doesn't know what the client considers unknown

Last year I did a contract job on a garden stake that monitored sunlight and soil hygrometer, that interfaced solely with the same company's smart sprinkler system. They used a binary json protocol but did evaluate xmpp, among other protocols, and determined that it would increase the cost of the stake by as much as 70% due to needing a larger solar cell to power it.

There's too much backlog to catch up on, but re: using the schema you can keep the pieces - yes, that's true of the schemas as provided, but I know that people produce their own schemas (or schema-like -things) based on the normalitive bits of the XEPs. Which is much less keep-both-parts.

(dwd: ^)

I agree GeOrG

Zash: the server would know that if the client uploads its schema, eg for EXI compression

Kev: that's only an argument about our schemas not being very well maintained, right?

Ge0rG: I think Dave was suggesting that people who do validation are on their own because our schemas aren't normative. I was drawing a distinction between doing validation using the normative text and using the provided non-normative schemas directly.

Our schemas aren't particularly well maintained, but I think that's an orthogonal issue to people who validate based on the bits that are well maintaned (the normative text).

The only schema that matters is the one support about the client, and that doesn't necessarily match the extensions the client supports. Also, clients (eg desktop) can always turn on non-strict encoding to support receiving unexpected XML

Kev: I thought that argument applies to any schema validation on xmpp and not just the official schema definitions

Ge0rG: AFAIK we have never said that it is e.g. valid to ignore MUSTs.

arc: I gather you're talking about EXI schemas here, rather than the general case that I started with.

We (or I) were originally talking about entities doing validity enforcement, particularly at security boundaries.

EXI schemas are XML schemas, because EXI is XML.

But there isn't any place stating that an XML stream MUST NOT have elements that aren't defined in a given indicated XEP

Ge0rG: No, indeed, but each XEP defines what is valid within its particular namespace.

I would fully support an extension that allows clients to define the XML schema they want to receive, EXI or not. I actually think that this would be better, especially if it was designed with EXI in mind.

Kev: yes, and the question right now is whether that definition is exhaustive by definition

Quite.

arc: that'd be awesome, provided that we can find an easy way to collect the full supported schema at compile time somehow

Note that XML schema informs EXI grammar, but these two things should not be confused. EXI grammar includes things like weighting certain elements that are seen more often in order to transfer them with fewer bits, etc.

arc: well, those parts of the grammar can only be obtained from prior measurements, right?

One of the major reasons peter's EXI XEPs are not workable is they confuse the two and do not define many aspects, eg, rules for how multiple schemas are combined to create a grammar.

GeOrG grammar could be developed through statistical analysis, but the EXI spec doesn't require any specific method. Only the client and server understand the grammar each other are using.

I would assume that in most cases software developers would design their own grammar.

arc: yeah, and then we need a mechanism to upload a grammar in a normal form and have a server side cache for grammars

Yup.

How much work is needed to get that into prototype state?

I have suggested a mechanism whereas the client assumes the server already has it, defines on the EXI stream by sha256, and then if the server does not support it responds with a "common" EXI grammar known to all clients with an error message that it does not have that grammar in cache, asking the client to restart and send the grammar first.

arc: yeah, but sha256 requires a normal form, and regular xml can be used if the switch fails

Another big problem with the existing EXI drafts is defining the schemas by URL that they can be fetched, on a connection before the user has authorized, allowing for a form of DDoS attack by having many clients contact the server requesting large files from third party hosts.

No, that'd require all clients to need support for plain text xml. There should be a standardized exi grammar for that transfer.

Removing plain text XML support almost has the size of the compiled library

Yeah, server side fetching of random URLs is a huge can of worms.

Remember that in order for this to be commercially viable, it has to work as well as proprietary alternatives. Squeezing a <1k binary grammar onto the library can be doable.

XXE is just one of the evil problems

Absolutely.

No idea if we can somehow make it secure nevertheless. Even if we require authentication beforehand, the vector is still there and needs to be mitigated

This is absolutely nothing with the work I've been doing over the last year, but I would happily participate in a working group to draft an extension for schema/grammar exchange and exi in general

Could have some set of centrally managed grammars maybe?

For my current project I'm kind of sad to be using Rust's xmpp parser crate

Simliar to the pre-made caps cache project that exists/-ed

Zash: that wouldn't scale, also schema can vary depending on which features a client has enabled

Zash I don't think there needs to be. For starters, it breaks the decentralized nature of xmpp. The code requirements to send the grammar is pretty small.

Ge0rG: I'm thinking Good Enough, not perfect for everyone. Something based on the Compliance Suite of the year maybe?

You're looking to solve a problem that doesn't exist. It's fine to have clients send their own grammar. What we lack is a standard grammar to transfer that grammar. And part of that, the schema for defining the grammar. That is outside of the scope of the exi standard.

EXI only requires that the grammar is understood by both parties. It does not even specify a standard grammar identifier, just the field in the exi header.

Zash: imagine a central authoritative directory of entity caps

That would be like that, but worse

arc: could we make a first attempt with oob transfer of grammars based on white listed URLs?

arc: is there any syntax available for grammars, outside of exi byte streams?

Again, I don't think it's necessary. In order to do that we need to have a standard format, and once we have a standard format we basically have the mechanism to send it from any client

Not sure what's so wrong about Zash's suggestion? A central catalogue of grammers seems like a good idea. It is not like you have to use it…

So you need to invent a grammar to transmit grammars.

Some libraries have a format they use, but as far as I've seen there is no standard.

flow: it will be useless

Ge0rG, that is what you have said before, but I still wonder why?

arc: how much work will it be? Are you actively working on it?

flow: because each compiled client artifact will have its unique grammar

At least if you want to use the grammar for server side filtering of unknown elements

No right now I am actively working on a H3 library in rust and a WebGL game engine. H3 (h3geo.org) is fascinating but unrelated to this.

Sounds a bit like ingress

No it is a plant scale MMO of a fictional earth-sized alien world. H3, being hierarchical and hexagon-based, is just an ideal solution. The engine only uses XMPP for authentication and to establish ICE for WebRTC audio and data channels.

https://youtu.be/wDuKeUkNLkQ?t=754 if you want to learn more, it's fascinating stuff and there is very good language support. Even a extension for PostGIS

Its been over a year since I've had any xmpp related contracts, but I'd be happy to contribute to a working group to get exi finally settled. If people want to work on it.

It sounds like a lot of the contract firms that used to support xmpp have lost interest, like &yet

arc: it's an interesting topic for me as a mobile client developer, but I have zero time to push it forward. The most I could do is writing s blog post or a wiki page outlining the status quo and the current challenges

I haven't been paying much attention because it's outside the scope of my work, but how is MIX progressed over the last year?

a bit, I guess we are at the state where we need more MIX implemenations to gather implementation experience. but besides an older (and potentially incomplete) ejabberd, there is only one, more recent one, in tigase. but hey, it's better than none :)

I think Daniel also has a MIX branch for conversations but I could imagine that this branch also stalled due the lack of MIX support in ejabberd

Someone reported recently that they have been working on a MIX plugin for Prosody

There's an old implementation in Openfire somewhere too

hmm hopefully that doesn't summarize the state of MIX: there is an old implemenation somehwere, maybe, but maybe not :)

Guus, There's the partial Surevine one in github.com/surevine/Openfire I think, yes.

Have not used it myself though

Guus, Though I also think that everyone involved with that has moved on from Surevine, in line with The Curse Of MIX.

There have reportedly been people moved on that had not even been involved with their MIX implementation. That's how bad The Curse is...

Surevine was afflicted by multiple curses. :-)

Is falling asleep every time I try to read the MIX XEPs part of the curse?

No, that's just a lack of attention span.

@board: I'm sadly unlikely to make the meeting later today

I'll try to lurk, but I've been pulled into a meeting

0. Welcome

Hi all. Welcome to the last Board meeting of this term.

Who do we have?

Well, not Guus.

:-(

o/

Quorum, yay.

Unless y'all disagree, I don't think there's much to discuss today.

Yep

Do you have something, MattJ?

Ok

Cool. Then, I'd like to thank everyone for their efforts this term!

Thank you for chairing :)

And good luck to the next one! :)

:-D

1. Date of Next

Tentatively +1W, with the newly elected Directors.

2. Close

Cheers!

Perfect, thank you!!

And I'm out my meeting

just in time, I see. 🙂

sorry for that.

“13:06:51 arc> For my current project I'm kind of sad to be using Rust's xmpp parser crate”, why sad?