jdev - 2021-11-03

on the topic of xml parsers

python has a xmlpullparser

but i think it will keep the whole xml document in memory

i think this is a problem .. i dont know how much data is exchanged between client and server, but i guess if a client runs for some days, that could be problematic

lovetox, smack uses an xml pull parser but splits the top level stream elements, so that at most one is kept in memory

also pull parser tend to not keep the whole document in memory

since you typically can't rewind, the unreachable parts of the document can be dropped

lovetox, xml.sax! :)

IIRC pull/push parsers can operate on streams, unlike DOM based parsers

of course, this may vary from parser implementation to parser implemention

having said that some xml parsers can operate on streams, I believe that in XMPP it's best to not parse on the stream but to split the top-level stream elements, as it, amongst other things, allows you to enforce a top-level stream element size limit

you can also enforce that with a parser which can tell you where in the stream it currently is

splitting sounds like implementing half on an XML parser which I definitely cannot recommend you doing

(I know that moparisthebest loves that kind of pain though :))

My personal experience is that the developing the splitter implmentation was not painful

but yes, it's basically an xml parser, or, i'd rather say, an xml state machine

and yes, if the parser is able to tell you the current position in the stream, then you probably do not need to do that

unfortunately, that was not true for in my case

We more or less use a streaming dom parser, by using a stream parser and then constructing a pseudo-dom on individual stanzas. I imagine we're not alone in that.

That's more or less what I do as well if I understood you correctly. It's all streaming until we need to do something with a stanza or an element, then the user has an option of parsing the whole element into a DOM or similar and operating on that instead of the streamed tokens.

jonas’, of course i can use sax, but then i have to write code to build the xml elements myself

or port to aioxmpp :-X

lovetox, actually, no, lxml.etree has a SAX -> etree thing

I use that in aioxmpp to capture unschematized elements

or used it anyway, I don't think I still do.

thats exactly what i need

lovetox, https://lxml.de/sax.html#building-a-tree-from-sax-events

then you just need a shim thing which handles the stream header and then delegates to that

https://lxml.de/parsing.html#modifying-the-tree

hurts to look at :)

yeah lxml build the same pullparser and added a method to delete elements when not needed anymore

I'd rather go the sax way where I can be sure of what's what

yea I tend to agree that sax parsers are painful, and it's better to split the stream into stanzas without parsing, and only parse individual stanzas as a whole/with a DOM parser if needed

i want a parser, that simply dispatches stanzas to my handlers. with that pullparser, i can just look for the "end" event, and at depth 1, i displatch the element that parser gives me.

thats a xml parser in probably 50 lines of code

lovetox, that sounds similar to what smack does

lovetox, and hope that the delete actually works as you wish :)

although what you really want to is something like jaxb, which matches the XML elements to types of your programming languages, with additional verification

jonas’, i will test this now

and also how vulnerable this parser is against attacks

so much possibilities to optimize the process, so little free time :) ✏

because there is zero settings here that i could configure to make it more secure

lovetox, this is a stand-alone file that splits a stream on stanza boundaries without an xml parser, so you'd get notified of individual stanzas https://github.com/moparisthebest/xmpp-proxy/blob/master/src/stanzafilter.rs#L255

yeah im not sure i should do this with python

its probably slow

but yeah i think this approach is cleaner

given that its a client application, I doubt that this particular overhead will be an issue

I wouldn't think it'd be slow in python, there aren't any string copies or anything, it's just running a switch on 1 character at a time, surely faster than an XML parser

but its additional, its not like i spare myself the xml parsing

you spare yourself the streaming sax parsing

Oo, maybe i missing something, all your filter is doing is cutting the stream into pieces and feeding it to the parser

oh you mean i dont need to care then about sax events

just give me the finished element after you parsed everything

that lxml lib is amazing, you can even register custom element classes, that will be used when it encounters a certain tag / namespace or event attribute combination

this thing is made for xmpp

lovetox funny, it's roughtly what I did for my XML library in Movim :D

https://github.com/movim/movim/tree/master/lib/moxl#payload

hm parsing 1 million xml elements and creating custom python classes for it, in 2 seconds

xml parsing is indeed not the bottleneck in a gui app

Want me to test on my server? o:)

In poezio it is very much the bottleneck.

Sadly.

why that?

Due to the way slixmpp converts DOM-like structs into XMPP-specific structs.

Link Mauve, please tell use more :)

maybe with an example code

https://lab.louiz.org/poezio/slixmpp/-/blob/master/slixmpp/xmlstream/stanzabase.py#L672 mostly.

This happens on each element['sub-element'] call, and is quite slow.

I once started to work on a JIT for that, but never managed to make poezio start.

what is the opinion on validating stanzas against a schema?

is this even possible?

can xml schemas be "open" in they allow elements not defined in the schema

and only validating MUST haves?

In my experience, XML Schema was too poor to be used for both validation and extraction of the data to be more machine-usable.

lovetox, yes, you can have <xs:any namespace='##other'/> in a schema.

I’ve been trying to fix our schemas every time I encounter something invalid or not restricted enough, but they clearly aren’t used for validation (or their users never contributed back their fixes).

In xmpp-parsers, I have drafted a DSL to represent the most common constructs we have in XMPP, but I translate manually from the XML Schema as well as from the XEP’s examples and text.

And I still can’t represent everything.

It made me discover a whole bunch of invalid things people do in the wild.

DSL eh?

And led to fixes in clients and servers.

I wonder if it maps to the OpenAPI JSON Schema XML stuff

Link Mauve, what i would like to do is just basic stuff like, checking that a presence type attribute is one of the allowed strings etc

right now i have to do this all in python, manually ✏

that would allow me later to have much cleaner code

and if not, what would you do?

i would only validate MUST

so invalid type on a presence, ignore stanza

or am i again to strict, and have to later add not failing on that because someone sends wrong values all the time :D

And if you already have that in Python, do you really gain much by having it in XML instead?

yes much cleaner code

now everywhere i have to be ready to catch exceptions, every attribute i query i have to be prepared for it to be None etc

parsing 4 attributes into enums, is then 30 lines of code

but true much is probably not possible to validate

many things have implicit defaults, so i need to check afterwards anyway if its there or not

lovetox, have a look at slixmpp or aioxmpp, they both already do so and in Python.

They validate, and also make a more ergonomic API.

I don't really see cleaner code. When you do pre-validation, you only safe the one "else" case for all invalid values which needs to throw an exception. In proper languages that's about one line of code, in python maybe three.

larma, catching obviously invalid stanzas early (and returning an error to the sender early) makes for cleaner code than doing it from random places all over the code

jonas’, I was thinking we parse as early as possible to not carry around heavy lifting strings when an enum value would've been enough...

yes, enumification is also a nice opportunity you get when doing things early ✏

Cleaner code... /me looks at https://hg.prosody.im/prosody-modules/file/58a20d5ac6e9/mod_rest/res/schema-xmpp.json

I made this monstrosity. Fear me!

lovetox, you can validate schema and should, if it's not obviously broken, with the one exception that XML schemas IIRC do not allow unspecified elements and attributes. but in XMPP we allow those everywhere (I believe some may disagree with me about that, but IMHO it's the only sane thing to do)

flow so how do i validate then? if someone adds a element, my validation fails ..

but its not invalid per xmpp definition

so basically you can validate XML for which an schema exists, but not XML for which no schema exists. those should be simply ignored, as you obviously know nothing about them and did not negogiate it either

but thats very extension unfriendly

well if a schema specifies the 'priority' attribute to be unsignedByte, you can validate that

i mean often XEPs start with someone just putting something additionally in there

flow i can only validate everything or nothing

not only one attribute

that was just an example ;)

> lovetox> flow i can only validate everything or nothing

I don't that that's true

In fact I know that this is not true :)

but maybe we are talking past each other

Does XML schema validators not ignore unknown things?

Zash, I don't think so no

that's the one thing where we in XMPP vary from how XML schemas are validated

IIRC you have to add some magic schema thingy in the elements schema to allow for arbitrary further child elements

that are not part of the schema

and IIRC we mostly don't do that in our XMPP schemas

lovetox, look for example at https://xmpp.org/extensions/xep-0203.html#schema

surely you can validate stamp

you can validate that it's value is in the correct format and that the required attribute actually exists

<delay xmlns='urn:xmpp:delay' from='juliet@capulet.com/balcony' stamp='2002-09-10T23:41:07Z'/>

but let's assume I add a child element into <delay/>

and you don't have a schema about it, it's purely optional and does not need to be negotiated

then you can still verify 'stamp', but not the child element ✏

so, it's not an all or nothing validation situation ✏

lovetox, does this help?

flow, it would help in that case to add an <xs:any namespace='##other'/> in the delay element.

sure, but let's just pretend that this is everywhere in our schemas

sprincle those everywhere 😕

Please no. ^^'

It makes validation (of the whole stanza) much harder.

what makes validation much harder?

In xmpp-parsers I add Option types for each known sub-element defined elsewhere, slixmpp doesn’t do that and instead keeps the extensibility everywhere, at the expense of performances.

flow, let’s say I want to assert that delay won’t be extended, or it is an error.

(For instance because I don’t support it to be extended.)

Leaving an extension point in it will take more memory and void the type safety of it.

While with the current schema, we can take it as OOP languages use the final keyword.

I think my general point is that in XMPP, unknown things should typically be simply ignored, as otherwise, the eXtensability in XMPP becomes unnecessary hard

servers can still filter unknown elements/attributes if they determined that those have not be negoiated between sender and recipient (but I am not sure if this is feasible)

unnecessary hard and even impossible in some situations

I’m ok with that in the generic public network case, but for specific implementations it does make sense to reject extensions you don’t know about.

Even if just for validation of other implementations.

Link Mauve> flow, let’s say I want to assert that delay won’t be extended, or it is an error. I may be misunderstanding the meaning of 'error' here, but I believe that mindest is harmful

it shouldn't be an error, it should simply be ignored

and it's fine if your native types don't allow the user to access those ignored elements/attributes

Link Mauve, define 'reject' here

Either ignore the whole delay element and only account for the other payloads, or ignore the whole stanza and reply with an error, stuff like that.

why would you ignore the whole delay element if it's perfectly fine otherwise? ✏

I would recommend more or less completely ignoring the schemas, personally.

when we say "validate", do we mean `validate(schema, stanza) : boolean` ?

if someone send you additional data within the delay element, without obviously negoiating it, then the sender has to assume that you may ignore it (because he doesn't know if you understand it)

Kev, in my personal experience, schemas in XEPs haven proven to be very helpful when implementing said XEP to clarify things that aren't clear from the text

Zash, I think so, yes, the stanza is either valid or not

What would a client do with this boolean?

the question is: what consitutes a valid stanza in the presence of unknown elements/attributes?

Fair enough. Having gone through some experience of people trying to use schemas for validating traffic, I feel confident in saying using them normatively can also lead to pain.

Sorry, something put an invalid delay tag in your message, so we threw the whole message into the trash

Zash, configurable, could be ignoring the stanza completely

Zash, define "invalid delay element" here?

Isn't what you actually want something that takes a schema and extracts the bits you care about, and ignores any undefined extras?

A data mapper, rather than a schema

Zash, what I do currently is `validate(schema, RFC-parts-of-the-stanza; foreach payload in stanza: validate(schema, payload)`.

is it like stamp='invalid'? or with an unknown child extension element?

Although that’s not completely correct as it is a transformative operation, not just a validation.

and I don't see how XML schema is particularly helpful for making such a thing

In the current example, I end up with a struct Delay { from: Jid, stamp: DateTime } for instance.

(And we *do* have a product that does validation of XMPP traffic across security boundaries, so I'm not saying "Don't do this", merely "Here there be dragons")

And I also agree with Kev, ignore schemas (or extract information useful to you) and write your own DSL.

Zash> Sorry, something put an invalid delay tag in your message, so we threw the whole message into the trash I think it is safer to throw the whole message in the trash, as ignoring parts of the elements of an stanza could modify the stanzas semantics

even though I am not aware of a concrete example

but I rather be defensive per default

I’d really like to have a server plugin to validate every stream, and report issues it finds to the developers of the relevant violating implementations.

flow: So a new XEP comes along and now you can't read any messages anymore because they have a new tag?

Zash, that's not what I had in mind

It's not even what I tried to express

I may be too tired today for properly reading anything, sorry if I misunderstand

trying to come up with a slightly related example

I read it that way too, fwiw *shrug*

I do this on the fly so it is maybe not good

if someone sends you an *invalid* stanza, e.g. an attribute which value 'foo' but it's specified to be an integer

I think then you can only ignore it

because the sender may wants to tell you something

Although I think it's been lost to the mists of time, we did have a DSL processor for Swiften to create the various parsers/serialisers/elements, but it wasn't XSD-based.

but you have no idea what, hence you can only guess what it is, and that leads to issues, because you may have guessed wrong

not treat it as if that child tag or extension was simply gone?

presense status with an invalid priority are probably a good example

flow, then you have a choice: either you work around it and still try to interpret it somehow, carry this tribal knowledge forever, and laugh at other implementations which didn’t make the same choice or carry it there too; or you report it to them so that they can stop sending the invalid value.

Zash, that's the tricky question

I’m firmly in the latter camp.

as I said, I could imagine that the semantic of a stanza is made up of multiple of its child elements

priority is a core thing, so uh

There's definitely, to my mind, a difference between unexpected, and expected-but-invalid content.

and then, if you ignore the existence of one of the child elements, the semantic may be different

Zash, yet I’ve received invalid ones, IIRC it was Gajim which didn’t validate what the user could input there.

It allowed numbers fewer than -128 or bigger than 127.

assume a client has set the highest priority 99999

he may then assumes that messages are routing according to this pirority

but servers may cap it?

Someone sent me a priority 500, my program rightfully rejected the stanza, I reported it and it got fixed.

or servers may treat it as 0?

So no one has to handle that case.

(flow, highest priority is 127.)

Link Mauve, I know :)

but the client wo set it to 99999 didn't

Right, so exactly my example. ^^

in the end, I believe it's the best for the open source community ecosystem of XMPP if implementations validate and reject invalid stanzas as a whole, and emit a visible error message in such a case, ideally pointing to the sending entity, with an request to report an issue to the vendor

+1

Link Mauve, :)

Link Mauve, btw, is there an equivalent of <xs:any namespace='##other'/> for attributes?

Yes, <xs:anyAttribute namespace="##other"/>.

We already use both in our XEPs.

But AFAIK we still disallow namespaced attributes.

There is still no XEP for an entity to advertise it properly supports XML namespaces.

not sure if we officially disallow them, it just seems that some seem to be afraid of them

Or that yeah.

We definitely have guidance not to use them, I'm just not sure where.