XSF Discussion - 2022-01-03

>This is about stopping XML libraries from _emitting_ anything that's outside our subset, no? Rather than the parsing being a problem. Yeah, those guys love doing things beyond what you want from them. I remember being shocked to find out that even though python has *four* xml libraries built in, every single one of them is vulnerable, and vulnerabilities differ between them. So Gajim for example can't use one of those. It even had to reconfigure expat somewhat IIRC. Now how often do you hear of a vulnerability in say json or bencode? Let alone those responsibilities being kept in a mainstream PL like they're part of functionality. <morning-rant> It's almost like xml just like java somehow inherently leads to overengineering. I recently learned that people have built several PL around xml alone, like xquery or xslt. What the actual hell? And more concretely take xmpp. Why come up with the *unique* format? They couldn't just use length prefixes even with that same xml for example, no they had to come up with a unique complicated xml-esque format you have to invent your own way of parsing. I mean I really want to believe they did have some different philosophy in mind. Maybe they intended for it to be parsed directly with SAX somehow? But that'd be no less complicated would it. Or take starttls. Why does it exist at all? Or srv ssl records. They couldn't just pick a standard ssl port like every other protocol does, of course, that's to simple for the xml mind. Or the mess that <message> generally is. When you receive it you have to work out if you're actually supposed to display it! I wonder what info at all that tag ends up conveying really. And then they come up with "message profiles", oh gosh... The XEP that especially shows is the "aesgcm" one. Where the guy pretty much says "screw this it's too complicated to add even a single xml tag" and he comes up with an adhoc markup format lmao. Other XEPs indicative of problems with XML are the various message text markup ones. XML is a freaking MARKUP language itslef yet people are so appalled by it they choose not to use it even in an XML based protocol. They either come up with their adhoc format, or that especially laughable XEP with " <span start="9" end="15"><emphasis/></span>". Like what the hell guys, what a perverted way with a markup language? This CANNOT be fully justified: either xml is problematic or the authors of the xep are (personally I think the former is the case) </morning-rant> >No, both sides are a problem. I remember an old (non XMPP, but similar issues) project where we ran into the <!proc exec cat /etc/passwd> or whatever it was issue. Turns out librarise out of the box were happy to do that, had no way to turn off all proc instances, and you had to know to turn it off. Exactly what I'm talking about. It's insane. Let alone it just doesn't belong in a *markup* language. Not that a markup language belongs as a data serialization language in the first place

Starttls exist for good reason, though I don't think it's useful these days. Back in the day though TLS and port numbers were expensive, so it was nice to be able to negotiate TLS over the same port you used for plain. SRV SSL records solve a real problem that other protocols that just pick a single port have too: sometimes even though there's a standard it's not on that port and you need service discovery.

Although I agree that starttls should just go away today and with basically everything about how XML has made things terribly complicated. The question is whether the tradeoff is worth it; eg. would it be *more* complicated to re-invent namespaces or some way of extending things in JSON where that's not just built in, for example.

I don't know what "message profiles" are though and have never really had any trouble figuring out if a message should be displayed, so I dunno about that one.

> Although I agree that starttls should just go away today and with basically everything about how XML has made things terribly complicated. The question is whether the tradeoff is worth it; eg. would it be *more* complicated to re-invent namespaces or some way of extending things in JSON where that's not just built in, for example. That is implying that namespaces are a necessary thing in the first place and not a pointless complication.

Yes, that is what I was implying, though I admit that I go back and forth on that. There may be better ways of extensibility.

kurisumakise: json parsing is equally full of vulnerabilities, I mean the most popular way to parse it in JavaScript is just to eval it

Turns out parsing untrusted input is hard

Comparing "not braindead eval'ing input" to "discarding a good number of parsers because in them the vulns cannot be fixed, picking that one parser and *configuring* it so that it's not vulnerable, also you have to build the object model on top of that yourself or come up with some hack" isn't exactly fair is it

kurisu: what about "those who don't learn from history are doomed to repeat it" https://www.moparisthebest.com/images/jsonschema.jpg

How many clients/servers actually end up verifying xml schemas? Are those even available for most xeps? Also I can't see how they're useful anyway. Perhaps if you're coding in a low level or extremely verbose language like java, xml schemas can be more expressive than the language itself, but otherwise all verification can easily be done in the main programming language you are using along with processing. ✏

kurisu, schemas are not just here for validation, they also tremendously help with understanding how the thing works. In xmpp-parsers I wrote the parsers often just from the schemas, and sometimes fixed the schemas alongside (when they didn’t match the text or the examples).

And yes, schemas are available for most XEPs.

I personally don’t mind XML, I don’t even see it most of the time, the parsers are doing the parsing for me, I only go down to the XML form when the parser throws an error.

And usually it’s the fault of the sending entity.

kurisu, you got my attention, do you have a pointer to a vuln in an xml parser that can be fixed?

> all verification can easily be done in the main programming language you are using along with processing You mean _languages_? In how many languages xmpp is implemented, again? If only there was a single language to describe formats and protocols...

Link Mauve, >chemas are not just here for validation, they also tremendously help with understanding how the thing works. In xmpp-parsers I wrote the parsers often just from the schemas, is that the "crate parsing common XMPP elements into Rust structures."? Thing is with a more or less sane format like json you don't need to create any structures, you can just work with json "directly", it readily maps nicely into you PL's structures if it's high level enough. A package like that just wouldn't need to exist if xmpp used a sane format. > sometimes fixed the schemas alongside lmao they don't even get those right. See how needlessly compilcated this is? >I personally don’t mind XML, I don’t even see it most of the time, the parsers are doing the parsing for me, I only go down to the XML form when the parser throws an error. you literally just described how you had to write a parser flow, indulge: https://docs.python.org/3/library/xml.html#xml-vulnerabilities Also if there are any gajim devs here please confirm or deny this: some time ago I remember digging thru nbxmpp source and seeing they actually had to change expat's option so that it wouldn't retrieve DTDs. Not a vuln but just shows you how overcomplicated xml as a goddamn serialization format is if using a standard configuration would deanonymize you - if it can retrieve dtds it means expat carries networking code, which I'm pretty certain would not magically pick up your proxy options. Also just try and imagine how much code is in that parser if it can retrieve stuff from the internet. This is called bloat, and bloat leads to vulns. Also I think Sam here even mentioned an arbitrary code exectuion vuln he stubmled upon in some code he was working with. Now I'm not even sure if the devs of that xml lib used even considered this a vuln not a feature. mjk, what do you even mean by this poor attempt at satire? Precisely my point, in every language you have to reinvent the parser to some degree because it's xml and moreover nonstandard xml. Are you refering to schemas being a single language for validation as a convenience? Well, no one uses that. In most clients it isn't even possible because they reinvent DOM, because xmpp is nonstandard xml. So to validate schemas most would have to reinventing an xml schema parser and validator which would take so, so much more lines of code then just validating the xml in you language. Even then if some client manage through some hacks to use a standard xml library with a validator included, how many of those do you think choose to go for xml schema validation and not native language validation? Apparently next to none at all, given that Link Mauve here had to fix the schemas As always, the xml/java/bloat mindset makes an issue out of a non-issue (validation) and thinks it's come up with a tool to solve it, and the tool ends up orders of magnitude harder to create, and probably even harder to use, than the original approach. Not even speaking of flexibility ✏

that table seems to say that xml parsers in python are maybe vuln to attacks that are based on features that we dont' use in XMPP and only if an outdated expat version is used

kurisu, in >20 years of XMPP you're not the first to complain about it using XML. You won't be the last. What are you hoping to achieve with this discussion, exactly?

> Are you refering to schemas being a single language for validation as a convenience? Yes. > Well, no one uses that. In most clients it isn't even possible because they reinvent DOM, because xmpp is nonstandard xml. I thought your argument was against xml schemas in general, I only used xmpp as an illustration of an xml format with wide cross-language implementation

and disabling those features is trivial (at least for java), see https://github.com/igniterealtime/Smack/blob/5f75d141ff6d0494f0e548d549be25c04440fe24/smack-xmlparser-stax/src/main/java/org/jivesoftware/smack/xml/stax/StaxXmlPullParserFactory.java#L36-L40

kurisu, no, JSON doesn’t “map nicely” to XMPP elements in any programming language, it is way too inexpressive for that.

kurisu, no, I used a ready-made XML parser, and parsed that in domain-specific structs.

Structs which are compile- and run-time checked for consistency, and which reject invalid states.

This avoids having to carry DOM around btw, the structs being specific to XMPP are much more concise in memory.

note that I'd also like if libraries would come with a secure default, I think at least some XML lib do, but I also believe there is a reason they don't (whether or not I agree with that reason is a different topic) ✏

yeah, smack does something similar

kurisu, also, you mention validators but in the wild it’s usually preferable to ignore unknown elements and extensions than to outright reject them.

Unless you have specific needs.

right, usually validation means, validating known elements, but ignoring unknown ones. only if you have specific needs, typically in form of increasing security by avoiding information leakage, you want to reject or strip unknowns

Link Mauve: do I understand correctly that there are, effectively, xml-schema→programming-language compilers that emit validation (and data conversion) code? ✏

mjk, there probably are, although I never used one.

Ah

I wrote xmpp-parsers to do this kind of validation, but using more domain-specific types than XML Schemas allow.

I see

So, you were the compiler :)

Sounds very similar to what ejabberd does.

XML Schema lists what is allowed where, with some typing, but they don’t encode enough for me to generate the parsers and serialisers just from that.

Yes, zinid and I had the same idea at roughly the same time. :)

mjk, not sure about the data conversation part, but there are XML parsers that you can hand a schema which will cause them to report schema violations ✏

mjk, not sure about the data conversion part, but there are XML parsers that you can hand a schema which will cause them to report schema violations ✏

MattJ: I also notice that in 20 years xmpp hasn't had many good clients developed for it. Even the major ones still are buggy. As much as I like the idea of a federated messenger, I'm starting to think there's some flaw inherent to the protocol that prevented it from taking off. I thought of trying to implement a client myself in my spare time. Well, it turns out the very serialization format is quite a stumbling block. So I thought maybe I'm just misunderstanding something, maybe e.g. I wasn't supposed to try and build a dom but e.g. work with sax events directly? More of curiousity at this point, as stated I'm just wondering what the original authors intended Link Mauve, >no, JSON doesn’t “map nicely” to XMPP elements in any programming language, it is way too inexpressive for that. You're being dishonest. XMPP is written in xml and you can't express xml more effeciently in json, true. However, the data conveyed by xmpp doesn't map nicely into xml in the first place. Take the list of mechanisms in stream:features. Not to mention how the xml format has no notion of a list. What does your library do with stream:features/mechanisms/mechanism/text()? Create whatever a list is called in rust? Well guess what, in a sane format like json it would already be a freaking list. Xml isn't to blame for not having a notion of a list, it's but a markup language afterall, not a data serialization one. It is however notable that even as a markup language it fails: xmpp doesn't use xml for markup like it was intended, it has to come up with an adhoc format in one xep and a laughable hack in another. So yes, unfortunately, because of the usage of xml you can't easily create a json version of xmpp. Because xml has its own object model of tree and nodes and attrs and whatnot on top of which actual structures like "a list of strings" have to be hacked on. This is in comparison to e.g. the torrent dht protocol: you could easily convert it to json or almost any other sane format because the initial format is sane. Because they don't use a markup language for data serialization. Because there are lists, dicts, strings ints and so on. Same goes for FCP, freenet's adhoc protocol. I've been developing something for freenet you see, and had to use said protocol, which didn't have a library for it in my language of choice. Well guess what it's been a freaking breeze to create my own parser. Funny how an adhoc,"afterthought" data serialization format is more pleasant to work with than xml >This avoids having to carry DOM around btw, the structs being specific to XMPP are much more concise in memory. Speaking of memory concise when speaking of xml.. ironic. >also, you mention validators but in the wild it’s usually preferable to ignore unknown elements and extensions than to outright reject them. and?

kurisu, given that things like whatsapp are based on XMPP, it's certainly not a flaw inherent in the protocol preventing it to be used for something useful.

it's just that nobody has made a business plan which supports the federated use case, so no moneys.

jonas': whatsapp has unlimited budget and thus manpower. With enough thrust even a pig will fly.

jonas’: well, that _is_ a flaw in the protocol, as viewed from the perspective of lock-in startups ;)

> there are XML parsers that you can hand a schema which will cause them to report schema violations That much I know. :) I was thinking more in the direction of compiling a single xml schema to byte- or machine code to produce a very efficient specialized validator/parser

Ge0rG, federation is not mandated by the protocol

jonas’: but client interop is

I also think that relatively young clients like Conversations and Dino demonstrate that it is possible to implement XMPP from scratch

kurisu, have a look at ActivityPub, I had the misfortune of digging into it, it’s JSON-LD all the way down.

I’ll take XMPP’s XML any day over that abomination.

You might like it better, maybe.

It does pretty much the same as XMPP, provides authentication, message passing, pubsub, etc.

Fun fact: there are no integers in JSON. ^^

Only numbers, which are double floats.

That makes it a natural fit for a Prosody module!.. Wait, Lua has integers now

I don't know why we are having this discussion, apart from ranting it will solve nothing and enlighten exactly nobody. The points being made are laughable to me for the most part (the lack of modern clients is that the "legacy" clients were good enough, until they weren't, and mobile-first with awful OSes and lack of proper networking were a big hurdle, that has nothing to do with XML, we had a good markup solution using XML elements 20 years ago already but I won't go back to this debate). The one I will acknowledge is that it is slightly harder to bootstrap a client from nothing, and that is mostly because it requires quite a bit of RFC diving and domain knowledge. But comparing XMPP to FCP or the torrent DHT protocol is disingenuous at best, considering the specialised and narrow target of what they are set to achieve.

+1

I have never used XML outside of XMPP, except for configuring openbox or writing web stuff, but I have seen complex data types and schemas in JSON and YAML and it certainly is not an pleasant sight.

(I'm not saying I don't like rants or ranting, but maybe there are other rooms that can better accommodate it than this room where people expect productive discussion, because no, XMPP is not switching to bencode tomorrow)

I am surely biased, but my impression is that the provided arguments against XMPP/XML are exaggerating the situation while additionaly describing it not precisely nor technically sound

And I don't buy the implication that there're obvious/simple alternatives. I.e. just avoid XML, reinvent everything from scratch and suddenly an extensible messaging spec becomes a simple, straightforward problem to solve, validation now is a non-issue and whatnot.

What I do buy is that JSON over HTTP does a much better job at on-boarding devs, being able to show a simple `curl` example for sending a message is a huge win.

seemingly atomic operations are a plus too

Seemingly atomic or seemingly a plus? :)

seemingly atomic

because any complex http+json protocol will have some kind of discovery / feature negotiation

Yeah

I still say that if we described XMPP has having a "JSX-like wire protocol" we'd be fine.

Anyway, the reality is that XMPP's biggest problems are that it's a connection-orientated protocol that connects clients. There's ways around both, but they require some effort to shift to a protocol that connects users with multiple clients. XML isn't really a problem, exce4pt that people perceive it to be one and get very hung up over it when they start.

I think the XML problem is much less problematic than people think it is, but only because XMPP really doesn't use XML as much as it uses some serialization and namespacing from XML. And that's very hard to explain, when people see XML and assume we're built on SOAP.

The "connection-orientated" problem is another conceptual problem, but also has technical problems. Mobile phones and suchlike simply cannot hold a connection open - a flaw in their design, but one we need to deal with. I've tried addressing this with XEP-0198 and persisting sessions beyond TCP connections, and I think that works, especially given the increasing need to persist a lot of information about clients in use. But that's met with resistance and I'm out of energy.

Similarly, the clients versus users problem; we have a series of half-solutions in that space too, but again, no energy.

And, of course, we're dealing with Matrix, which has a nice set of shiny clients, and a marketing/sales team happy to distort the truth as much as they can, which is stealing away a lot of market share.

addressing both persistent connections AND clients v users is so overwhelming that it sometimes feels like we might as well start over

Oh, note I didn't say "persistent connections", I said "connection-orientated". They're somewhat different.

And clients v users feels achingly close to solved, some days. In fairness, it's easier to go in this direction than the other - a lot of effort on IMAP went on going from user to multiple clients, and it was *hard*.

The other thing we could do - and I gave it some thought a while back - would be to define a server-side client and a standardized traditional HTTP API (whether graph or RESTish).

That's somewhat akin to what Lloyd Watkin did a few years back, mind, but perhaps taking things a bit further.

dwd, mod_rest in Prosody is already kind of like that I think.

Minus the standardisation.

huh

so

there is that idea of tracking devices

if we were to track devices and had device-specific tokens which are good for, say, a day or so.

and which can be renewed via a proper connection with SASL

and those token would, while valid, represent a session with smacks and push and whatnot and could be used with a mod_rest-ish interface

that would be like 95% of what's needed to resolve the connection-oriented stuff, right?

and it doesn't seem that complex

I mean, tracking devices and per-device auth is a bit of work, but there are people interested in solving that anyway (I think Snikket is, even if only for proper multi-device onboarding and management)

I definitely wouldn't say "if only for" - I think the server having no knowledge of a user's clients is one of the big things holding us back right now

We do, sort of, have this knowledge already (XEP-0198, push registrations) but it's incomplete and fragmented

CSS of the wiki pages does not look right on my phone using Chrome https://wiki.xmpp.org/web/Main_Page

dwd: > And, of course, we're dealing with Matrix, which has a nice set of shiny clients, and a marketing/sales team happy to distort the truth as much as they can, which is stealing away a lot of market share. Sales team? How do they make money?

kurisu: it's created by a VC funded company that sells consulting services and maintains the only usable clients

and hosting

Sam: someone pays for matrix related consulting? What sort of consulting I wonder

Anything similar seem happening with xmpp?

> Anything similar seem happening with xmpp? That's how a good amount of the people working within the XSF make a living

See https://xmpp.work/all-listings/ for example

> Daniel escribió: > See https://xmpp.work/all-listings/ for example Btw reminder to everyone here to place make their entries as I plan to advertise sooner or later

emus, Process One are missing, you may want to reach out to them directly

yes ok thanks!

any contact recommendations?

emus: you could try the ejabberd muc

jonas’: oh surez totally forgot

kurisu, Lots of consulting, but also Element have a bunch of pay-for addons. Most of what Matrix is being used for is bridging, as far as I can see. But their Teams and WhatsApp bridges are not open source, but per-seat licensing.

Sam, You're wrong, of course. Matrix is maintained by a foundation which is entirely seperate from Element, and just happens to have the same staff, infrastructure, and accounts.

Heh, oh yes, silly me, how could I ever forget

In any case, I've been told by people that XMPP can't do picture messaging, for example. They've been told this by Matrix/Element sales staff. Of course, the Matrix bridge is curated carefully to prevent anything nice from working, so it appears this way, but...

Also matrix invented the concept of bridging, nothing else had this decades before

I feel severely insulted as the developer of the matrix client with the longest code history.

dwd: I'm in a MUC has bridge with Matrix and Discord, I can download files but can't open them. Is that intended?

John: bridged by a bot, or using bifröst?

wgreenhouse: I'm not sure, I think bot, I can see the Matrix users as participants in the MUC, like in other transports

If you can download, and not open, its a problem of your OS isn't it?

Menel: I tried to download using Conversations. You can try https://matrix.andrzejszczepaniak.co.uk/_matrix/media/v1/download/t2bot.io/15c36c5778f007732c5b98139069f93a6a8e91f9

Conversations is delegating it to my webbrowser and is opening fine ✏

Worked on my browser too

John: that doesn't show as a button, or "oob" stanza per xep-0067

which is the req. for conversations to open ✏

wgreenhouse: it shows as a button, maybe OOB

if it shows as a button but Conversations won't open it, hook Conversations up to adb and maybe report a bug

sounds like the bridge is trying to oob but something goes wrong

ordinary bifrost rooms do have working picture attachments

when they work at all

I will update Conversations to make sure first

Maranda has been doing some work on the bridge lately

https://aria-net.org/_matrix/media/v1/download/aria-net.org/dDNSPVQaDKXIhSHgQKvIckvD

🤔🤔

Maranda: I see the link as a plain text, no download button

That depends on mime / size and C related configuration iirc

I don't recall issues with the oob element in the stanza itself, but if that's the case just give me a poke and I'll fix what runs on aria-net.org at least

No OOB there.

Presumably because of the module that stops you from posting pictures here.

https://aria-net.org/_matrix/media/v1/download/aria-net.org/eTSToHRtaxBpIxLPcApGfbgl

Indeed

That's a Premium member feature

Ge0rG: to be able to sent pictures? It is the bridge you are paying?

msavoritias: posting pictures in this room is only allowed to members since The Incident. The "premium" part was just a bit of irony about membership being paid in most places (but not here)

Ah. Totally missed the joke on my part :p I just assumed it to be a matrix closed feature out of habit

moparisthebest: > Also matrix invented the concept of bridging, nothing else had this decades before Matrix bridges are well integrated into clients. Now to use xmpp bridges to just irc I had to actually ask about working instances on biboumi chat (b/c you can't just google them, let alone connect automatically), then also find out which ones work lmao. I think we should give credit where credit's due

kurisu: I fail to see the links between your three statements

Try being honest with yourself and you will

I haven't dabbled in matrix clients in a while so I will ignore the first, the second is more about having a SPOF with a humongous IRC bridge on matrix.org, as far as I understand it?

Aren't matrix bridges practically centralized? In which case there's nothig to discover

(I am not saying it does not work, or that the matrix IRC bridging is not a success, although I find it to be a bit abrasive from the IRC side, but I fail to see how it is relevant)

kurisu: are you saying the matrix clients have the IRC gateways hardcoded? Doesn't sound very federated to me

kurisu: https://www.moparisthebest.com/images/xmpp-vs-matrix.jpg

moparisthebest: > kurisu: are you saying the matrix clients have the IRC gateways hardcoded? Doesn't sound very federated to me I don't know, I only know that it just werks And no, builtin gateways don't harm federation. At the end of the day Conversations by default offers its server, is that a problem now? moparisthebest: > kurisu: https://www.moparisthebest.com/images/xmpp-vs-matrix.jpg I wish that honesty were the case irl.