XSF Discussion - 2021-09-03

following a comment on HN, someone suggested that the entire protocol be versioned to allow easier guessing of features https://news.ycombinator.com/item?id=28393085

maybe that's a bit extreme, but advertising Compliance Suite version as part of disco could be an idea?

so that your server/client could inform you that a newer client/server may provide a better experience, maybe?

I don't see any gains to be had from that, only worse interop and pointlessly nagging users

We already have disco on a feature-by-feature basis, so there's no need for a "global" one (i.e. advertise compliance suites)

Erm but compliance suite is just an informative documento even not an actual feature, that's misleading ✏

fair enough :)

Displaying Compliance Suites on xmpp.org's Software page is already controversial :)

I thought it was just not completed and merged yet

Yeah, it would be a good addition.

I think the controversy was about client developers being inclined to lie about it, and nobody from the XSF being able to prove

(or disprove)

isn't there a compliance checker like compliance.conversations.im for Complaince Suites(yet)?

what would it check?

whether all specs mentioned in the compliance suite are implemented correctly by a specific client/server

ahahahhaha

*ahem*

no

testing servers is hard, testing clients is harder.

Testing the tester isn't trivial either

a tool to check any server/client for correct implementation would be great

note that compliance.c.im also does not what you say.

i'm a big fan of specification-driven development in principle: https://ttm.sh/2l3.md

but, likely impossible

too long and badly typeset, didn't raed.

> moparisthebest> a tool to check any server/client for correct implementation would be great I've never heard anyone say this before /s

moparisthebest, well, the aioxmpp test suite has found a surprising amount of bugs ;)

... in servers.

Zash, yes but the tester is a single implementation that compares others, so if it raises questionable results about clients which appear to work, it's easy to detect and fix

"easy" - off you go ;)

i think we've had this discussion in the past but i'd be curious to integrate Scansion in markdown codeblocks to be part of the specs (although it's only for server testing so far, right?)

MattJ, having a single tester impl say "all clients are broken" is easier to detect/debug than random client/server combinations acting up :P (arguably)

kindof like doctests

The easy way is to look at the DOAP and hope it's not a pack of blatant lies

jonas’, yeah exactly i love doctests from python/rust and i'd be curious how it could be useful for a protocol spec

southerntofu, if you think it's easy, I encourage you to try it. It's a lot of work, and complicated.

`echo echo BROKEN > compliance-checker.sh`

southerntofu, so, markdown isn't really a format used for XMPP specs to start with ...

The Editor is speaking

jonas’, could be XML for all i care, i personally like markdown :)

I personally too like markdown.

Meanwhile everyone is using Markdown

one day we should migrate XEPs to use markdown

it would make a lot of things easier.

and more accessible.

it would also make other things harder.

anyway

different sujbect

southerntofu, so, actually, I really like your idea

XMPP flavored markdown

the doctest-style tests in XMPP specs.

Whatnow?

it would be interesting to see something like that for a simple spec like '30 and a non-trivial spec like '45

yea they should be written in XEP-0393 instead

/kickban moparisthebest

servers are easy to automate

clients... not so much

jonas’, i mention multiple (de)serialization formats in "# Other spectest-compliant formats" on that draft blogpost

southerntofu, as I said ... "too long and badly typeset, didn't read."

XEP-compliant XML could be one of those

what jonas’ said

It's Friday, not Readlongmarkdownonbacklitscreenday

but in the end, the format is irrelevant

the hard part is actually running the tests

against servers it's doable

but clients... no clue

e.g. see scansion

Zash, TLDR

exactly, scansion

> As we've seen in the previous articles, open standards and associated test suites are key to achieving expected results, and therefore benefit the whole ecosystem. But often, these two concerns are treated as separate problems to deal with in entirely different ways. As a result, some specifications are so long and complex that coming up with a test suite is a challenge (XMPP/ActivityPub), while other systems are backed by a collection of unspecified test suites that are hard to comprehend (Ansible).

or aioxmpp test suite

> Could we get the best of both worlds by treating specification and compliance (testing) as a single problem? This hypothetical approach i call specification-driven development, whereby a specification document is intended both for human and machine consumption. In that case, the specification contains a written presentation of concepts, in addition to a machine-readable test suite that follows a certain format to programmatically ensure that the concepts and behavior described in the specification are implemented properly. This format for specifications is called a spectest document. (TODO: maybe specdoc ?)

needs more paragraphs

if you can write a specification such that a program can create tests out of it, then a program could also just implement it correctly for you instead ?

no programmer needed

moparisthebest, see also IDL

moparisthebest, the tests would be handcrafted *as part of the spec*

Accept: application/epub+zip delivered onto my e-ink device (it has wifi disabled). Good luck!

southerntofu, any practical idea on how to test clients though?

I don't actually think that helps in 99% of cases southerntofu

moparisthebest, I think it does.

in many ways

jonas’, i imagine a client could be intrsumented by running in a "test mode" where it exposes a well-known API (over a socket or something) instructing it to react in a certain manner?

imagine all examples in a XEP could actually be run against servers.

the specs are usually clear, and it's easy to test the things in the XEP

southerntofu, ideally without changing the code of existing clients

that doesn't help with the 99999 things that actually happen in practice

moparisthebest, I disagree.

`gajim-control` you say?

> a tool to check any server/client for correct implementation would be great uhm at Tigase we have https://github.com/tigase/tigase-tts-ng (and older https://github.com/tigase/tigase-testsuite) which do just that; probably could be nice to generalise it and make available as a service...

don't get me wrong, it'd be very helpful, a great start, but you'd still miss a lot of things

I know :-)

moparisthebest, if each XEP contained machine tests in addition to human formats, it would help for two things: it would help to clear ambiguities before reaching implementation stage (because if the test contradits human text then somebody will notice earlier and raise issues) and it would help somewhat-correct implementation because you could just do ./xeptest XEP-0030.xml -- myclient --test-mode

moparisthebest, of course, it will never be perfect, but perfect is the enemy of good.

Zash, i'm not aware of gajim-control

jonas’, without changing client code i don't think is possible? would be much harder to emulate user interactions on every possible client :) :)

we already have "examples are not the spec" surely "tests are not the spec" would be that way too, so I don't think it helps with ambiguities

moparisthebest, but tests would actually be normative

but maybe the test protocol can be simple enough for all clients to implement?

jonas’, yes exactly, no ambiguity in there :)

southerntofu, I have no clue how that protocol would look like

except if you accidentally XMPP the protocol.

but that's kind of not the goal

like, if you instruct a client to add a roster item .... and such ... you end up respecifying XMPP

to instruct a client to do XMPP

yeah but on a higher-level i guess

maybe, but only slightly

it needs to be pretty fine grained if you want to do things like testing individual MUC interactions

and in fact maybe it can serve as guidelines for library UX? so that if the specs are well-written you could just follow the test API and that would be your client library?

I'm not even aware of tools to do non-browser UI testing

("grant voice to user", "kick user", "unban user"...)

moparisthebest, they exist

they typically use a11y APIs

I'm sure they do, just never had the opportunity I guess :)

i mean i guess all clients/libraries have a "mute" or "ban" function, maybe standardizing that could be part of the answer?

then it's just a matter of serializing/deserializing instructions to that "API"

southerntofu, I don't see where that could be standardized really

especially across languages.

and UI paradigms

but that standardisation ... it would be like XMPP itself.

then we could have per-XEP testing interface where we expect you to expose some glue to your client code for specific functions?

what are you actually after testing though

southerntofu, same thing really

the client author doesn't want to implement a whole other xmpp protocol to see if his "join muc" works, he wants to click "join muc" and see if that works

jonas’, well per-spec is much easier to implement/maintain/test

southerntofu, doesn't solve the core issue though

and like i said if you're developing a client/library from scratch, it can serve as implementation guideline for making a good library API

it is another XMPP-ish thing besides XMPP

yup

for libraries, the story is slightly different and we are slowly venturing toward the lands of IDLs

IDLs?

interface description languages

("interface" as in API)

ah cool that's sort of what i was talking about :P

like OpenGL publishing an XML file where the entire API is contained so that you can auto-generate headers for any language

but that approach only really works (a) in one single lagnuage/paradigm or (b) for really low-level things

yeah it wouldn't work for UI testing most likely, but personally i'm fine with that

then you just have library testing I guess

if we reach server & library testing that'd already be a huge milestone :)

anyway we'll see, when i find a complete week to hack on something and see where it goes

southerntofu, feel free to loop me (the XEP Editor) in.

can scansion support multiple clients on different servers to keep S2S?

to test s2S*

I know of no reason why it couldn't

but does it already?

like can i run two scansion instances with the same "scansion script" where one is client A and one is cliebt B and they're aware of each other's address? and the test can be run on two accounts on the same server (simple e2e test) or across servers (account for s2s bugs)?

fro my reading of the homepage it does but i'd like to be sure: https://matthewwild.co.uk/projects/scansion/

Tried to catch up on the context, but for my library I wrote https://pkg.go.dev/mellium.im/xmpp/internal/integration which is a framework for spinning up servers and clients and linking them up to test against. Works very well for my own integration tests (except for ejabberd which never shuts down cleanly for some reason, so right now it's only running tests that require a server against prosody) and could in theory be adapted to work for other languages and the like and be more stand alone.

southerntofu, one scansion script can describe multiple clients, which could likely connect to different servers, but that's not something I've tested

Many of the ones we use to test Prosody involve multiple actors interacting

ok i'll make sure to test that then :)

Lots of examples in https://hg.prosody.im/trunk/file/tip/spec/scansion/

I imagine changing `jid: juliet@localhost` to @example.com etc would work.

yeah that was the gist of my question :)

so it looks like there's at least three different testing frameworks for XMPP lcients/servers (tigase/mellium/scansion)

Time to make a testing framework compliance tester!

would be interesting to see how they compare and whether they could be reunited as part of a single spec

Zash, maybe not that far haha :P

i mean tigase and mellium appear (from a quick look) to be both object oriented testing APIs.. if they could have the same interface it *could* be easier to maintain a single testing library with bindings to a bunch of supported languages)

Write a scansion script parser in Go and Java?

yeah wy not, or in any language with C FFI, or..

i don't know what the best answer is, but i know there's possibilities to explore

The problem is that most libraries try to abstract the protocol, which is different to testing the protocol. You'll only test that the library author's model and assumptions hold true.

That's why Scansion is low-level (but easy), but also tests are typically server-dependent.

why would test be server-dependent? because the server would return more information if it supports further extensions?

or because the fields in a stanza would not be serialized in the same order?

Scansion ignores additional elements (in unexpected namespaces) by default

Also element order in most cases iirc (there are some different checking modes)

But there are multiple ways to implement something and be compliant with the spec in many places

Every time you read "MAY" in a spec, that's a gain for implementers and a pain for testers :)

MattJ, surely there's a *finite* number of ways to comply with the spec, right? if so "OR Juliet receives" (boolean logic) could do the trick?

You could copy the script and have one for each possibility. The challenge is not collapsing into a black hole.

The finite possibilities quickly add up

I'm not saying it's impossible, I'm just saying it's a lot of work

(although like you said scansion as it is would be ill-suited to test clients/libraries, maybe a unified test API would be more suited in that case)

I'm also not saying it's not valuable work, but nobody is motivated to do it after they've already implemented (to their knowledge) a correct XMPP implementation

MattJ, i do understand the possibility of that in an extensible protoco but in regards to XMPP i don't see, do you have any practical example that comes to mind?

i've found so far that specs were rather complete or ambiguous but never offered infinite interpretations

I'm not claiming any offer infinite interpretations

(contrary to ActivityPub/Microformats for instance)

(or that none do)

I encourage you to just try it if you're interested in solving this problem

sure, i'll certainly try with scansion to test multiple servers

in the meantime i'd be curious how Wojtek and Sam see each other's testing framework and whether they could imagine to cooperate in a hypothetical future on a single testing API/suite

It might be better to have different testing implementations

I think it's more valuable to have multiple, especially if real implementations are built on the same libraries

well then cooperate on a test format that could be implemented by both testing systems?

A test output format would be nice, to say "I tested XEP-nnnn support, it passed"

Aren't there a pile of such formats?

Hey there, I've just been hit by https://docs.python.org/3/whatsnew/3.9.html#urllib-parse (Python 3.9 URL parsing does not recognise ";" anymore as a separator, following W3C changes). As we use ";" as a separator in XMPP URIs, that smells like a more general issue for us

I've had to work around the Android Java URI parser not recognizing that some years ago already.

python has a new parameter, so the fix should be easy there

Mmmm, URI vs URL fun.

More like URFd

Ge0rG, southerntofu, Zash, wurstsalat: but we host the xmpp providers list, which tries to reference each entry