southerntofufollowing a comment on HN, someone suggested that the entire protocol be versioned to allow easier guessing of features https://news.ycombinator.com/item?id=28393085
southerntofumaybe that's a bit extreme, but advertising Compliance Suite version as part of disco could be an idea?
southerntofuso that your server/client could inform you that a newer client/server may provide a better experience, maybe?
MattJI don't see any gains to be had from that, only worse interop and pointlessly nagging users
MattJWe already have disco on a feature-by-feature basis, so there's no need for a "global" one (i.e. advertise compliance suites)
MarandaErm but compliance suite is just an informative documento even not an actual feature, that's misleading even✎
MarandaErm but compliance suite is just an informative documento even not an actual feature, that's misleading ✏
southerntofufair enough :)
wurstsalatDisplaying Compliance Suites on xmpp.org's Software page is already controversial :)
ZashI thought it was just not completed and merged yet
Ge0rGYeah, it would be a good addition.
Ge0rGI think the controversy was about client developers being inclined to lie about it, and nobody from the XSF being able to prove
southerntofuisn't there a compliance checker like compliance.conversations.im for Complaince Suites(yet)?
jonas’what would it check?
southerntofuwhether all specs mentioned in the compliance suite are implemented correctly by a specific client/server
jonas’testing servers is hard, testing clients is harder.
ZashTesting the tester isn't trivial either
moparisthebesta tool to check any server/client for correct implementation would be great
jonas’note that compliance.c.im also does not what you say.
southerntofui'm a big fan of specification-driven development in principle: https://ttm.sh/2l3.md
moparisthebestbut, likely impossible
jonas’too long and badly typeset, didn't raed.
MattJ> moparisthebest> a tool to check any server/client for correct implementation would be great
I've never heard anyone say this before /s
jonas’moparisthebest, well, the aioxmpp test suite has found a surprising amount of bugs ;)
jonas’... in servers.
southerntofuZash, yes but the tester is a single implementation that compares others, so if it raises questionable results about clients which appear to work, it's easy to detect and fix
MattJ"easy" - off you go ;)
southerntofui think we've had this discussion in the past but i'd be curious to integrate Scansion in markdown codeblocks to be part of the specs (although it's only for server testing so far, right?)
southerntofuMattJ, having a single tester impl say "all clients are broken" is easier to detect/debug than random client/server combinations acting up :P (arguably)
jonas’kindof like doctests
ZashThe easy way is to look at the DOAP and hope it's not a pack of blatant lies
southerntofujonas’, yeah exactly i love doctests from python/rust and i'd be curious how it could be useful for a protocol spec
MattJsoutherntofu, if you think it's easy, I encourage you to try it. It's a lot of work, and complicated.
Zash`echo echo BROKEN > compliance-checker.sh`
jonas’southerntofu, so, markdown isn't really a format used for XMPP specs to start with ...
MattJThe Editor is speaking
southerntofujonas’, could be XML for all i care, i personally like markdown :)
jonas’I personally too like markdown.
MattJMeanwhile everyone is using Markdown
jonas’one day we should migrate XEPs to use markdown
jonas’it would make a lot of things easier.
jonas’and more accessible.
jonas’it would also make other things harder.
jonas’southerntofu, so, actually, I really like your idea
ZashXMPP flavored markdown
jonas’the doctest-style tests in XMPP specs.
jonas’it would be interesting to see something like that for a simple spec like '30 and a non-trivial spec like '45
moparisthebestyea they should be written in XEP-0393 instead
jonas’servers are easy to automate
jonas’clients... not so much
southerntofujonas’, i mention multiple (de)serialization formats in "# Other spectest-compliant formats" on that draft blogpost
jonas’southerntofu, as I said ... "too long and badly typeset, didn't read."
southerntofuXEP-compliant XML could be one of those
Zashwhat jonas’ said
ZashIt's Friday, not Readlongmarkdownonbacklitscreenday
jonas’but in the end, the format is irrelevant
jonas’the hard part is actually running the tests
jonas’against servers it's doable
jonas’but clients... no clue
Zashe.g. see scansion
southerntofu> As we've seen in the previous articles, open standards and associated test suites are key to achieving expected results, and therefore benefit the whole ecosystem. But often, these two concerns are treated as separate problems to deal with in entirely different ways. As a result, some specifications are so long and complex that coming up with a test suite is a challenge (XMPP/ActivityPub), while other systems are backed by a collection of unspecified test suites that are hard to comprehend (Ansible).
jonas’or aioxmpp test suite
southerntofu> Could we get the best of both worlds by treating specification and compliance (testing) as a single problem? This hypothetical approach i call specification-driven development, whereby a specification document is intended both for human and machine consumption. In that case, the specification contains a written presentation of concepts, in addition to a machine-readable test suite that follows a certain format to programmatically ensure that the concepts and behavior described in the specification are implemented properly. This format for specifications is called a spectest document. (TODO: maybe specdoc ?)
jonas’needs more paragraphs
moparisthebestif you can write a specification such that a program can create tests out of it, then a program could also just implement it correctly for you instead ?
moparisthebestno programmer needed
jonas’moparisthebest, see also IDL
southerntofumoparisthebest, the tests would be handcrafted *as part of the spec*
ZashAccept: application/epub+zip delivered onto my e-ink device (it has wifi disabled). Good luck!
jonas’southerntofu, any practical idea on how to test clients though?
moparisthebestI don't actually think that helps in 99% of cases southerntofu
jonas’moparisthebest, I think it does.
jonas’in many ways
southerntofujonas’, i imagine a client could be intrsumented by running in a "test mode" where it exposes a well-known API (over a socket or something) instructing it to react in a certain manner?
jonas’imagine all examples in a XEP could actually be run against servers.
moparisthebestthe specs are usually clear, and it's easy to test the things in the XEP
jonas’southerntofu, ideally without changing the code of existing clients
moparisthebestthat doesn't help with the 99999 things that actually happen in practice
jonas’moparisthebest, I disagree.
Zash`gajim-control` you say?
Wojtek> a tool to check any server/client for correct implementation would be great
uhm at Tigase we have https://github.com/tigase/tigase-tts-ng (and older https://github.com/tigase/tigase-testsuite) which do just that; probably could be nice to generalise it and make available as a service...
moparisthebestdon't get me wrong, it'd be very helpful, a great start, but you'd still miss a lot of things
WojtekI know :-)
southerntofumoparisthebest, if each XEP contained machine tests in addition to human formats, it would help for two things: it would help to clear ambiguities before reaching implementation stage (because if the test contradits human text then somebody will notice earlier and raise issues) and it would help somewhat-correct implementation because you could just do ./xeptest XEP-0030.xml -- myclient --test-mode
jonas’moparisthebest, of course, it will never be perfect, but perfect is the enemy of good.
southerntofuZash, i'm not aware of gajim-control
southerntofujonas’, without changing client code i don't think is possible? would be much harder to emulate user interactions on every possible client :) :)
moparisthebestwe already have "examples are not the spec" surely "tests are not the spec" would be that way too, so I don't think it helps with ambiguities
jonas’moparisthebest, but tests would actually be normative
southerntofubut maybe the test protocol can be simple enough for all clients to implement?
southerntofujonas’, yes exactly, no ambiguity in there :)
jonas’southerntofu, I have no clue how that protocol would look like
jonas’except if you accidentally XMPP the protocol.
jonas’but that's kind of not the goal
jonas’like, if you instruct a client to add a roster item .... and such ... you end up respecifying XMPP
jonas’to instruct a client to do XMPP
southerntofuyeah but on a higher-level i guess
jonas’maybe, but only slightly
jonas’it needs to be pretty fine grained if you want to do things like testing individual MUC interactions
southerntofuand in fact maybe it can serve as guidelines for library UX? so that if the specs are well-written you could just follow the test API and that would be your client library?
moparisthebestI'm not even aware of tools to do non-browser UI testing
jonas’("grant voice to user", "kick user", "unban user"...)
jonas’moparisthebest, they exist
jonas’they typically use a11y APIs
moparisthebestI'm sure they do, just never had the opportunity I guess :)
southerntofui mean i guess all clients/libraries have a "mute" or "ban" function, maybe standardizing that could be part of the answer?
southerntofuthen it's just a matter of serializing/deserializing instructions to that "API"
jonas’southerntofu, I don't see where that could be standardized really
jonas’especially across languages.
jonas’and UI paradigms
jonas’but that standardisation ... it would be like XMPP itself.
southerntofuthen we could have per-XEP testing interface where we expect you to expose some glue to your client code for specific functions?
moparisthebestwhat are you actually after testing though
jonas’southerntofu, same thing really
moparisthebestthe client author doesn't want to implement a whole other xmpp protocol to see if his "join muc" works, he wants to click "join muc" and see if that works
southerntofujonas’, well per-spec is much easier to implement/maintain/test
jonas’southerntofu, doesn't solve the core issue though
southerntofuand like i said if you're developing a client/library from scratch, it can serve as implementation guideline for making a good library API
jonas’it is another XMPP-ish thing besides XMPP
jonas’for libraries, the story is slightly different and we are slowly venturing toward the lands of IDLs
jonas’interface description languages
jonas’("interface" as in API)
southerntofuah cool that's sort of what i was talking about :P
jonas’like OpenGL publishing an XML file where the entire API is contained so that you can auto-generate headers for any language
jonas’but that approach only really works (a) in one single lagnuage/paradigm or (b) for really low-level things
southerntofuyeah it wouldn't work for UI testing most likely, but personally i'm fine with that
moparisthebestthen you just have library testing I guess
southerntofuif we reach server & library testing that'd already be a huge milestone :)
southerntofuanyway we'll see, when i find a complete week to hack on something and see where it goes
jonas’southerntofu, feel free to loop me (the XEP Editor) in.
southerntofucan scansion support multiple clients on different servers to keep S2S?
southerntofuto test s2S*
ZashI know of no reason why it couldn't
southerntofubut does it already?
southerntofulike can i run two scansion instances with the same "scansion script" where one is client A and one is cliebt B and they're aware of each other's address? and the test can be run on two accounts on the same server (simple e2e test) or across servers (account for s2s bugs)?
southerntofufro my reading of the homepage it does but i'd like to be sure: https://matthewwild.co.uk/projects/scansion/
SamTried to catch up on the context, but for my library I wrote https://pkg.go.dev/mellium.im/xmpp/internal/integration which is a framework for spinning up servers and clients and linking them up to test against. Works very well for my own integration tests (except for ejabberd which never shuts down cleanly for some reason, so right now it's only running tests that require a server against prosody) and could in theory be adapted to work for other languages and the like and be more stand alone.
Zashsoutherntofu, one scansion script can describe multiple clients, which could likely connect to different servers, but that's not something I've tested
ZashMany of the ones we use to test Prosody involve multiple actors interacting
southerntofuok i'll make sure to test that then :)
ZashLots of examples in https://hg.prosody.im/trunk/file/tip/spec/scansion/
ZashI imagine changing `jid: juliet@localhost` to @example.com etc would work.
southerntofuyeah that was the gist of my question :)
southerntofuso it looks like there's at least three different testing frameworks for XMPP lcients/servers (tigase/mellium/scansion)
ZashTime to make a testing framework compliance tester!
southerntofuwould be interesting to see how they compare and whether they could be reunited as part of a single spec
southerntofuZash, maybe not that far haha :P
southerntofui mean tigase and mellium appear (from a quick look) to be both object oriented testing APIs.. if they could have the same interface it *could* be easier to maintain a single testing library with bindings to a bunch of supported languages)
ZashWrite a scansion script parser in Go and Java?
southerntofuyeah wy not, or in any language with C FFI, or..
southerntofui don't know what the best answer is, but i know there's possibilities to explore
MattJThe problem is that most libraries try to abstract the protocol, which is different to testing the protocol. You'll only test that the library author's model and assumptions hold true.
MattJThat's why Scansion is low-level (but easy), but also tests are typically server-dependent.
southerntofuwhy would test be server-dependent? because the server would return more information if it supports further extensions?
southerntofuor because the fields in a stanza would not be serialized in the same order?
MattJScansion ignores additional elements (in unexpected namespaces) by default
MattJAlso element order in most cases iirc (there are some different checking modes)
MattJBut there are multiple ways to implement something and be compliant with the spec in many places
MattJEvery time you read "MAY" in a spec, that's a gain for implementers and a pain for testers :)
southerntofuMattJ, surely there's a *finite* number of ways to comply with the spec, right? if so "OR Juliet receives" (boolean logic) could do the trick?
ZashYou could copy the script and have one for each possibility. The challenge is not collapsing into a black hole.
MattJThe finite possibilities quickly add up
MattJI'm not saying it's impossible, I'm just saying it's a lot of work
southerntofu(although like you said scansion as it is would be ill-suited to test clients/libraries, maybe a unified test API would be more suited in that case)
MattJI'm also not saying it's not valuable work, but nobody is motivated to do it after they've already implemented (to their knowledge) a correct XMPP implementation
southerntofuMattJ, i do understand the possibility of that in an extensible protoco but in regards to XMPP i don't see, do you have any practical example that comes to mind?
southerntofui've found so far that specs were rather complete or ambiguous but never offered infinite interpretations
MattJI'm not claiming any offer infinite interpretations
southerntofu(contrary to ActivityPub/Microformats for instance)
MattJ(or that none do)
MattJI encourage you to just try it if you're interested in solving this problem
southerntofusure, i'll certainly try with scansion to test multiple servers
southerntofuin the meantime i'd be curious how Wojtek and Sam see each other's testing framework and whether they could imagine to cooperate in a hypothetical future on a single testing API/suite
moparisthebestIt might be better to have different testing implementations
MattJI think it's more valuable to have multiple, especially if real implementations are built on the same libraries
southerntofuwell then cooperate on a test format that could be implemented by both testing systems?
MattJA test output format would be nice, to say "I tested XEP-nnnn support, it passed"
ZashAren't there a pile of such formats?
goffiHey there, I've just been hit by https://docs.python.org/3/whatsnew/3.9.html#urllib-parse (Python 3.9 URL parsing does not recognise ";" anymore as a separator, following W3C changes). As we use ";" as a separator in XMPP
URIs, that smells like a more general issue for us
Ge0rGI've had to work around the Android Java URI parser not recognizing that some years ago already.
goffipython has a new parameter, so the fix should be easy there
ZashMmmm, URI vs URL fun.
moparisthebestMore like URFd
emusGe0rG, southerntofu, Zash, wurstsalat:
but we host the xmpp providers list, which tries to reference each entry