-
Daniel
flow, > This PR introduces a breaking change of the wire protocol and hence requires a namespace bump. that's not stricly true for experimental XEPs though, right?
-
jonas’
yes
-
Daniel
i mean whether or not it's a good idea to bump can be up to debate. but there is no rule that says you have to
-
Daniel
(and in that particular case I don’t think it's a good idea to bump)
-
Zash
What now?
-
Daniel
https://github.com/xsf/xeps/pull/998
-
flow
Daniel, every xep with a number should undergo a namespace bump if a breakking wire protocol change is introduced
-
Daniel
flow, according to your opinion or according to some rule?
-
jonas’
flow, quote?
-
Zash
Can a thing implementing the old version interop with something implementing the new version?
-
flow
I think that is what was practiced in the past (look for example at the jingle namespace, I'd assume a large part was done while the xep was experimental)
-
flow
Zash, in this case the answer is no
-
Daniel
the fact that individual authors of MAM and jingle decided to bump doesn’t mean that's always a good idea
-
flow
Daniel, I think it is always a good idea as soon as the xep becomes a number
-
Daniel
speaking broadly here (looking a bit at MAM) I as a client author often find it easier to just look for a different element or look out for the subtle differences between protocol versions instead implementing a whole new namespace. that might be due to how my library is set up
-
Daniel
but code for "is attribute x here if not behave slighlty different" is often less than replicating objects
-
flow
One of our objectives is to "ensure interoperability", I think we would violate that objective if we did breaking changes to the wire protocol without a namespace bump in official XEPs
-
Ge0rG
> whether or not it's a good idea to bump can be up to debate. It is never a good idea to bump.
-
Daniel
anyhow in the particular case of 420+omemo; nobody implements the current version
-
Daniel
so we end up increasing the namespace version to infinity without practical use
-
flow
Daniel, that is an argument that an namespace bump does not incur a high cost in this case
-
Daniel
flow, we should ensure interop for anything that is Draft+
-
larma
(in which case we don't namespace bump, because namespace bump is not interoperable)
-
flow
Daniel, I think we should ensure interop for every standard we adopted, but we should introduce a supervised place for ProtoXEPs where things can be changed without a nemsapce bump
-
Daniel
that's called 'experimental'
-
flow
I think we are talking more or less about the same thing: we need a place where XEPs can be easily modified without a namespace bump, but I think if experimental becomes that place, then it would eventually hurd our reputation
-
flow
"look XMPP is heavily modular and now even the single modules don't interoperate"
-
Daniel
experimental doesn’t become that place it is that place
-
Daniel
has been for 20 years
-
Ge0rG
flow: you remember that time when we tried to bump Carbons?
-
larma
I'd guess that most developers that implement experimental XEPs are well aware that they might have breaking changes...
-
flow
that's a guess. I for example wouldn't expect that
-
larma
isn't there a bold warning above experimental XEPs, telling they are only for exploratory implementations and not production?
-
flow
larma, there is, but that doesn't imply that after you determined via disco#info lookup that an remote entity supports the xep, it actually doesn't
-
Ge0rG
with the number of Experimental XEPs running in production, people must be ignoring those warnings
-
Daniel
I think that's a failure on our part that we haven’t been moving XEP ups the chain fast enough
-
flow
again, yes I also want to have a curated place for ProtoXEPs which can undergo breaking changes without a namespace bump, but experimental isn't the place
-
Daniel
a failure that we have slowly been trying to correct
-
Ge0rG
flow: for most protocol changes a namespace bump is just the wrong tool. A new feature is much more elegant and well feasible for many protocol changes.
-
flow
could be a simple as adding protoxeps/ to the xsf repo and render the XEPs within there
-
flow
Ge0rG, if you can do it via a new feature then this is obviously the superior approach to a namespace bump
-
flow
Ge0rG, but sadly I don't think this is possible in the PR in question
-
Ge0rG
flow: I remember this same discussion from a year or two ago, and I'm pretty sure the conclusion was that all the tools are in place and we are just using Experimental and Draft wrong.
-
Daniel
i'm not entirely sure that we reached a conclusion. but at least that wasn’t an unpopular opinion
-
Ge0rG
well, there are two parties involved in "using XEP status", the author who might want to push a XEP forward, and the Council that ultimately makes the decision
-
Ge0rG
we even introduced Document Shepherds who can push a XEP forward on behalf of the community.
-
Ge0rG
but I don't see wide use of that yet.
-
Ge0rG
Also community feedback to LCs and CFEs is... sparse
-
larma
(yesterday it was discussed in council what happens when file-metadata needs a namespace bump. answer is: it doesn't. All elements are optional and adding new optional elements isn't backwards incompatible. Old implementations will just ignore the new elements. That's good enough for both experimental and draft status. In Final, new elements can be added via new XEPs by using a new namespace. So I'm not aiming to ever namespace bump that XEP.)
-
flow
larma, yes, if I understand the case with file-metadata correctly, this wouldn't require a namespace bump: since all elements are optional, interop is easy, even if new optional elements are added this wouldn't require a namespace bump. only if new required elements are added
-
Ge0rG
larma: what if you want to / have to rename elements?
-
flow
Ge0rG, I guess it depends if interop would become an issue in that case
-
larma
Beside that, we have clear rules on namespace bumps in an Active, Procedural XEP, so it's not like something for council to decide on. https://xmpp.org/extensions/xep-0053.html: > While the XEP is in the Experimental state, if appropriate and in consultation with the author(s), the XMPP Registrar shall update the namespace version number at the end of namespace (e.g., from "0" to "1"); the XMPP Registrar shall do so only if the protocol defined in the XEP has been modified in a way that is not backwards-compatible with an earlier version of the protocol.
-
larma
Ge0rG, I hardly see how renaming would ever be needed...
-
Ge0rG
larma: well, that was a rather hypothetical question indeed
-
Kev
> All elements are optional and adding new optional elements isn't backwards incompatible. That's not entirely true. Someone can write a validator that ensures that the payloads match the allowed protocol (and people do this), so adding new bits of allowed protocol is not backwards compatible with such implementations.
-
eevvoor
I have been banned from the kuketzblog muc without having written there anything. Does anyone else has the same problem?
-
flow
Kev, I don't like that line of thinking. That would go against the "extensible" in XMPP
-
flow
A validator can only validate what he knows, not what he doesn't know
-
flow
And we shouldn't close the door on introducing new optional protocol parts without having to bump the namespace
-
Kev
flow: Extensible in XMPP has always meant in practice that you can add new namespaces, not that you can change what is allowable within a single namespace.
-
flow
Kev, i would challenge that in two ways: first, i am not sure if this is true in the past, and secondly, I think it shouldn't be true in the present and future
-
Kev
Whether you like it or not, I'm asserting that this is how the specs are used. Not typically on Internet deployments, but very much in other deployments people care about such things :)
-
flow
in any way, this should be documented
-
flow
but I really don't like the idea what we can't even introduce new optional attributes without having a namespace bump
-
Kev
flow: We have had such discussions in the past, and I have 99% sure came to the conclusion I've described above, FWIW. And I do not disagree with documenting it.
-
Kev
We can still add new attributes, FWIW, but by negotiation.
-
Kev
We just can't send them unilaterally and expect that the result will be accepted.
-
Ge0rG
Kev: but that effectively means that we can only add new attributes by version bump, not by new feature vars
-
Ge0rG
because that would break strict parsers not knowing about the new feature
-
flow
Ge0rG, I assume validators validate the structure and content kind, not the concrete content type
-
larma
Kev, you'd say it needs to be specifically written in the XEP that more elements are allowed? Because if it is written, the validating entity is actually misbehaving by rejecting unknown elements ;)
- Ge0rG meant "elements", not "attributes" in the above statement, but it's actually intersting to know for both
-
larma
"Future versions of this XEP may add new elements. Entities are therefor required to silently ignore all unknown elements."
-
Ge0rG
larma: it'd be *very* interesting for EXI and other systems that require full knowledge of all supported elements
-
flow
Ge0rG, how?
-
Kev
larma: Sorry, I was picking up the general point, not a particular XEP. If it says that, all is good.
-
larma
It doesn't say it explicitly *yet*, but I'm open on adding it 😉
-
flow
i feel this would probably lead to more and more future XEPs adding this a disclaimer, not sure if we shouldn't simply switch the default and have XEPs state that they won't add new things without a namespace bump
-
eevvoor
Is there some netiquette when to ban people?
-
larma
flow, I agree on that at least for experimental XEPs. For Draft XEPs it might be not that appropriate: given that drafts include a schema, it should be possible to actually use that schema for validating/exi usage (although I personally think our current exi spec is overly complex and would be more likely to be actually implemented if severely simplified such that it's not doing any schema-based compression at all, but that's a different story)
-
flow
larma, EXI can cope with incomplete schemas, that's not an issue. And XEP schemas describe that is there, not what is not there
-
larma
actually I've seen many schemas describing what is not there...
-
flow
larma, possibly, but given that we all aggree that one can plug in new elements under a different namespace (nearly) everywhere in xmpp…
-
flow
larma, actually do you have an example for that?
-
larma
https://xmpp.org/extensions/xep-0060.html item element
-
larma
It has explicit "<xs:any namespace='##other'/>"
-
flow
is that surprising given that <item/> is a container element for arbitrary payload elements?
-
larma
No, but still it explicitly describes what should be default
-
Kev
I'm not convinced that moving to a world where you can't write validators because anything goes is an optimal direction. Maybe an Experimental/Draft split there would help, but equally that would work if you consider Experimental to not be deployed, and if you consider Experimental not to be deployed then namespace bumps are also not harmful.
-
Kev
But I don't have bandwidth to give it much thought this morning.
-
Zash
Should https://xmpp.org/extensions/xep-0353.html still be in LC?
-
jonas’
2019
-
jonas’
no.
-
jonas’
I intend on '0001-restarting those expired LCs after the next elections
-
dwd
Kev, I think we have, in the past, assumed that adding an optional attribute in an existing namespace is OK. Whether we still should is another matter.
-
Ge0rG
dwd: what about adding new elements in an existing namespace?
-
dwd
Ge0rG, I can't think of an example of that.
-
dwd
Ge0rG, Whereas, for adding an optional attribute, we did that in XEP-0237 (later folded into RFC 6121).
-
Ge0rG
Is there a principal difference between adding elements and attributes, wrt strict schema validators?
-
dwd
Kev, FWIW, my understanding is that we have consistently said that if you use the schemas to validate traffic at runtime, you get to keep the pieces. As I say, we may want to change this, but it would be, I think, a change, rather than "moving to a world".
-
dwd
Ge0rG, I don't know of one. I think it might be marginally easier to deal with unexpected attributes when processing XML in code than unexpected elements, and we do assume that certain elements only contain a single child.
-
Ge0rG
dwd: I'd say that adding unknown elements is a problem orthogonal to adding multiple elements where only one is allowed
-
Ge0rG
I assume you are looking at the allowed content for IQ stanzas
-
dwd
Ge0rG, Probably, yes, but it might explain why we've avoided it, explicitly or not.
-
dwd
Ge0rG, I was indeed thinking that, and the question-mark over multiple elements within the XEP-0060 <item/> element.
-
Ge0rG
One could argue that the semantic impact of unknown attributes is smaller than of unknown elements
-
dwd
Yes, and Kev notes that negotiation makes all this a lot safer.
-
dwd
Though I think XEP-0115 added attributes without negotiation.
-
Ge0rG
Last year we had a very interesting what-if discussion in the context of steam compression and EXI, which I'd summarize as "a given client should exactly know the schema it can process, and could communicate that to the server on login, so that a custom minimal optimized exi or compression lookup table can be used for that connection, stripping all unknown elements / attributes at the server side "
-
dwd
Yup, I recall that. Lead mostly by arc I think.
-
arc
Yup
-
arc
That allows simple clients with hardwired EXI, such as microcontrollers
-
arc
Simple client, complex server is a common xmpp protocol design principle
-
dwd
Ge0rG, But I don't think that a client requesting that unknown-to-it attributes are stripped by the server is breaking the end-to-end principle.
-
Ge0rG
dwd: given that the client would ignore those attributes anyway, yes. But the same is true for elements
-
dwd
Ge0rG, Yes, it is. But I think we developed our unwritten rules of thumb around hand-parsing XML rather than anything else.
-
arc
I agree with GeOrG, there's no purpose sending XML schema elements to clients which are not designed to understand them.
-
dwd
(And again: I'm not saying that our unwritten rules of thumb are right for today's developers)
-
dwd
arc, s/not designed/explicitly claim not/
-
arc
Sure it's *nice* to have a xml console showing the raw traffic. That is an edge case that clients can ask for with EXI.
-
Ge0rG
dwd: so what does that mean for writing down our unwritten rules of thumb?
-
Ge0rG
Should Next Council create an Informational XEP about how and when to bump namespaces?
-
Zash
If one is submitted, Council will vote on it.
-
jonas’
I seem to recall to have proposed to do that and it got shot down
-
jonas’
because "it is obvious anyway"
-
jonas’
despite such discussions coming up again and again
-
Zash
If it's obvious then it should be easy to write it down
-
Ge0rG
jonas’: I don't remember your offer or it being shut down, but I think it's a good idea, especially if it covers schema validators and EXI
-
arc
EXI literally has a flag to turn on/off whether you want strict schema or not. The problem of servers sending clients unexpected xml elements is one of bandwidth. The sender does not, and should not, know what bandwidth restrictions exist on the receiving side. For example, some IoT devices may turn off support for regular message body, ie chat. They might work only on pubsub and adhoc commands, and regular chat messages can cause their tiny battery powered cpu to wake up too often for them to run on tiny solar cell power.
-
Ge0rG
arc: the same mechanism would help mobile clients. No need to send chat state notifications and wake up the device if the client can't display them anyway
-
Ge0rG
So by stripping all unknown elements, then dropping empty containers, you'd get immediate optimization
-
Zash
Would have been nice if "supports normal chat messages" was in disco/caps
-
Zash
... 20 years ago
-
Zash
Ge0rG, tricky if the server doesn't know what the client considers unknown
-
arc
Last year I did a contract job on a garden stake that monitored sunlight and soil hygrometer, that interfaced solely with the same company's smart sprinkler system. They used a binary json protocol but did evaluate xmpp, among other protocols, and determined that it would increase the cost of the stake by as much as 70% due to needing a larger solar cell to power it.
-
Kev
There's too much backlog to catch up on, but re: using the schema you can keep the pieces - yes, that's true of the schemas as provided, but I know that people produce their own schemas (or schema-like -things) based on the normalitive bits of the XEPs. Which is much less keep-both-parts.
-
Kev
(dwd: ^)
-
arc
I agree GeOrG
-
Ge0rG
Zash: the server would know that if the client uploads its schema, eg for EXI compression
-
Ge0rG
Kev: that's only an argument about our schemas not being very well maintained, right?
-
Kev
Ge0rG: I think Dave was suggesting that people who do validation are on their own because our schemas aren't normative. I was drawing a distinction between doing validation using the normative text and using the provided non-normative schemas directly.
-
Kev
Our schemas aren't particularly well maintained, but I think that's an orthogonal issue to people who validate based on the bits that are well maintaned (the normative text).
-
arc
The only schema that matters is the one support about the client, and that doesn't necessarily match the extensions the client supports. Also, clients (eg desktop) can always turn on non-strict encoding to support receiving unexpected XML
-
Ge0rG
Kev: I thought that argument applies to any schema validation on xmpp and not just the official schema definitions
-
Kev
Ge0rG: AFAIK we have never said that it is e.g. valid to ignore MUSTs.
-
Kev
arc: I gather you're talking about EXI schemas here, rather than the general case that I started with.
-
Kev
We (or I) were originally talking about entities doing validity enforcement, particularly at security boundaries.
-
arc
EXI schemas are XML schemas, because EXI is XML.
-
Ge0rG
But there isn't any place stating that an XML stream MUST NOT have elements that aren't defined in a given indicated XEP
-
Kev
Ge0rG: No, indeed, but each XEP defines what is valid within its particular namespace.
-
arc
I would fully support an extension that allows clients to define the XML schema they want to receive, EXI or not. I actually think that this would be better, especially if it was designed with EXI in mind.
-
Ge0rG
Kev: yes, and the question right now is whether that definition is exhaustive by definition
-
Kev
Quite.
-
Ge0rG
arc: that'd be awesome, provided that we can find an easy way to collect the full supported schema at compile time somehow
-
arc
Note that XML schema informs EXI grammar, but these two things should not be confused. EXI grammar includes things like weighting certain elements that are seen more often in order to transfer them with fewer bits, etc.
-
Ge0rG
arc: well, those parts of the grammar can only be obtained from prior measurements, right?
-
arc
One of the major reasons peter's EXI XEPs are not workable is they confuse the two and do not define many aspects, eg, rules for how multiple schemas are combined to create a grammar.
-
arc
GeOrG grammar could be developed through statistical analysis, but the EXI spec doesn't require any specific method. Only the client and server understand the grammar each other are using.
-
arc
I would assume that in most cases software developers would design their own grammar.
-
Ge0rG
arc: yeah, and then we need a mechanism to upload a grammar in a normal form and have a server side cache for grammars
-
arc
Yup.
-
Ge0rG
How much work is needed to get that into prototype state?
-
arc
I have suggested a mechanism whereas the client assumes the server already has it, defines on the EXI stream by sha256, and then if the server does not support it responds with a "common" EXI grammar known to all clients with an error message that it does not have that grammar in cache, asking the client to restart and send the grammar first.
-
Ge0rG
arc: yeah, but sha256 requires a normal form, and regular xml can be used if the switch fails
-
arc
Another big problem with the existing EXI drafts is defining the schemas by URL that they can be fetched, on a connection before the user has authorized, allowing for a form of DDoS attack by having many clients contact the server requesting large files from third party hosts.
-
arc
No, that'd require all clients to need support for plain text xml. There should be a standardized exi grammar for that transfer.
-
arc
Removing plain text XML support almost has the size of the compiled library
-
Ge0rG
Yeah, server side fetching of random URLs is a huge can of worms.
-
arc
Remember that in order for this to be commercially viable, it has to work as well as proprietary alternatives. Squeezing a <1k binary grammar onto the library can be doable.
-
Ge0rG
XXE is just one of the evil problems
-
arc
Absolutely.
-
Ge0rG
No idea if we can somehow make it secure nevertheless. Even if we require authentication beforehand, the vector is still there and needs to be mitigated
-
arc
This is absolutely nothing with the work I've been doing over the last year, but I would happily participate in a working group to draft an extension for schema/grammar exchange and exi in general
-
Zash
Could have some set of centrally managed grammars maybe?
-
arc
For my current project I'm kind of sad to be using Rust's xmpp parser crate
-
Zash
Simliar to the pre-made caps cache project that exists/-ed
-
Ge0rG
Zash: that wouldn't scale, also schema can vary depending on which features a client has enabled
-
arc
Zash I don't think there needs to be. For starters, it breaks the decentralized nature of xmpp. The code requirements to send the grammar is pretty small.
-
Zash
Ge0rG: I'm thinking Good Enough, not perfect for everyone. Something based on the Compliance Suite of the year maybe?
-
arc
You're looking to solve a problem that doesn't exist. It's fine to have clients send their own grammar. What we lack is a standard grammar to transfer that grammar. And part of that, the schema for defining the grammar. That is outside of the scope of the exi standard.
-
arc
EXI only requires that the grammar is understood by both parties. It does not even specify a standard grammar identifier, just the field in the exi header.
-
Ge0rG
Zash: imagine a central authoritative directory of entity caps
-
Ge0rG
That would be like that, but worse
-
Ge0rG
arc: could we make a first attempt with oob transfer of grammars based on white listed URLs?
-
Ge0rG
arc: is there any syntax available for grammars, outside of exi byte streams?
-
arc
Again, I don't think it's necessary. In order to do that we need to have a standard format, and once we have a standard format we basically have the mechanism to send it from any client
-
flow
Not sure what's so wrong about Zash's suggestion? A central catalogue of grammers seems like a good idea. It is not like you have to use it…
-
Ge0rG
So you need to invent a grammar to transmit grammars.
-
arc
Some libraries have a format they use, but as far as I've seen there is no standard.
-
Ge0rG
flow: it will be useless
-
flow
Ge0rG, that is what you have said before, but I still wonder why?
-
Ge0rG
arc: how much work will it be? Are you actively working on it?
-
Ge0rG
flow: because each compiled client artifact will have its unique grammar
-
Ge0rG
At least if you want to use the grammar for server side filtering of unknown elements
-
arc
No right now I am actively working on a H3 library in rust and a WebGL game engine. H3 (h3geo.org) is fascinating but unrelated to this.
-
Ge0rG
Sounds a bit like ingress
-
arc
No it is a plant scale MMO of a fictional earth-sized alien world. H3, being hierarchical and hexagon-based, is just an ideal solution. The engine only uses XMPP for authentication and to establish ICE for WebRTC audio and data channels.
-
arc
https://youtu.be/wDuKeUkNLkQ?t=754 if you want to learn more, it's fascinating stuff and there is very good language support. Even a extension for PostGIS
-
arc
Its been over a year since I've had any xmpp related contracts, but I'd be happy to contribute to a working group to get exi finally settled. If people want to work on it.
-
arc
It sounds like a lot of the contract firms that used to support xmpp have lost interest, like &yet
-
Ge0rG
arc: it's an interesting topic for me as a mobile client developer, but I have zero time to push it forward. The most I could do is writing s blog post or a wiki page outlining the status quo and the current challenges
-
arc
I haven't been paying much attention because it's outside the scope of my work, but how is MIX progressed over the last year?
-
flow
a bit, I guess we are at the state where we need more MIX implemenations to gather implementation experience. but besides an older (and potentially incomplete) ejabberd, there is only one, more recent one, in tigase. but hey, it's better than none :)
-
flow
I think Daniel also has a MIX branch for conversations but I could imagine that this branch also stalled due the lack of MIX support in ejabberd
-
MattJ
Someone reported recently that they have been working on a MIX plugin for Prosody
-
Guus
There's an old implementation in Openfire somewhere too
-
flow
hmm hopefully that doesn't summarize the state of MIX: there is an old implemenation somehwere, maybe, but maybe not :)
-
dwd
Guus, There's the partial Surevine one in github.com/surevine/Openfire I think, yes.
-
Guus
Have not used it myself though
-
dwd
Guus, Though I also think that everyone involved with that has moved on from Surevine, in line with The Curse Of MIX.
-
Guus
There have reportedly been people moved on that had not even been involved with their MIX implementation. That's how bad The Curse is...
-
dwd
Surevine was afflicted by multiple curses. :-)
-
Zash
Is falling asleep every time I try to read the MIX XEPs part of the curse?
-
Guus
No, that's just a lack of attention span.
-
Guus
@board: I'm sadly unlikely to make the meeting later today
-
Guus
I'll try to lurk, but I've been pulled into a meeting
- ralphm bangs gavel
-
ralphm
0. Welcome
-
ralphm
Hi all. Welcome to the last Board meeting of this term.
-
ralphm
Who do we have?
-
dwd
Well, not Guus.
-
ralphm
:-(
- Seve says hi!
-
MattJ
o/
-
ralphm
Quorum, yay.
-
ralphm
Unless y'all disagree, I don't think there's much to discuss today.
- Seve agrees.
-
MattJ
Yep
-
Seve
Do you have something, MattJ?
-
Seve
Ok
-
ralphm
Cool. Then, I'd like to thank everyone for their efforts this term!
-
MattJ
Thank you for chairing :)
-
Seve
And good luck to the next one! :)
-
ralphm
:-D
-
ralphm
1. Date of Next
-
ralphm
Tentatively +1W, with the newly elected Directors.
-
ralphm
2. Close
-
ralphm
Cheers!
-
Seve
Perfect, thank you!!
- ralphm bangs gavel
-
Guus
And I'm out my meeting
-
Guus
just in time, I see. 🙂
-
Guus
sorry for that.
-
Link Mauve
“13:06:51 arc> For my current project I'm kind of sad to be using Rust's xmpp parser crate”, why sad?