XSF Discussion - 2021-09-03

  1. southerntofu

    following a comment on HN, someone suggested that the entire protocol be versioned to allow easier guessing of features https://news.ycombinator.com/item?id=28393085

  2. southerntofu

    maybe that's a bit extreme, but advertising Compliance Suite version as part of disco could be an idea?

  3. southerntofu

    so that your server/client could inform you that a newer client/server may provide a better experience, maybe?

  4. MattJ

    I don't see any gains to be had from that, only worse interop and pointlessly nagging users

  5. MattJ

    We already have disco on a feature-by-feature basis, so there's no need for a "global" one (i.e. advertise compliance suites)

  6. Maranda

    Erm but compliance suite is just an informative documento even not an actual feature, that's misleading even

  7. Maranda

    Erm but compliance suite is just an informative documento even not an actual feature, that's misleading

  8. southerntofu

    fair enough :)

  9. wurstsalat

    Displaying Compliance Suites on xmpp.org's Software page is already controversial :)

  10. Zash

    I thought it was just not completed and merged yet

  11. Ge0rG

    Yeah, it would be a good addition.

  12. Ge0rG

    I think the controversy was about client developers being inclined to lie about it, and nobody from the XSF being able to prove

  13. Ge0rG

    (or disprove)

  14. southerntofu

    isn't there a compliance checker like compliance.conversations.im for Complaince Suites(yet)?

  15. jonas’

    what would it check?

  16. southerntofu

    whether all specs mentioned in the compliance suite are implemented correctly by a specific client/server

  17. jonas’


  18. jonas’


  19. jonas’


  20. jonas’

    testing servers is hard, testing clients is harder.

  21. Zash

    Testing the tester isn't trivial either

  22. moparisthebest

    a tool to check any server/client for correct implementation would be great

  23. jonas’

    note that compliance.c.im also does not what you say.

  24. southerntofu

    i'm a big fan of specification-driven development in principle: https://ttm.sh/2l3.md

  25. moparisthebest

    but, likely impossible

  26. jonas’

    too long and badly typeset, didn't raed.

  27. MattJ

    > moparisthebest> a tool to check any server/client for correct implementation would be great I've never heard anyone say this before /s

  28. jonas’

    moparisthebest, well, the aioxmpp test suite has found a surprising amount of bugs ;)

  29. jonas’

    ... in servers.

  30. southerntofu

    Zash, yes but the tester is a single implementation that compares others, so if it raises questionable results about clients which appear to work, it's easy to detect and fix

  31. MattJ

    "easy" - off you go ;)

  32. southerntofu

    i think we've had this discussion in the past but i'd be curious to integrate Scansion in markdown codeblocks to be part of the specs (although it's only for server testing so far, right?)

  33. southerntofu

    MattJ, having a single tester impl say "all clients are broken" is easier to detect/debug than random client/server combinations acting up :P (arguably)

  34. jonas’

    kindof like doctests

  35. Zash

    The easy way is to look at the DOAP and hope it's not a pack of blatant lies

  36. southerntofu

    jonas’, yeah exactly i love doctests from python/rust and i'd be curious how it could be useful for a protocol spec

  37. MattJ

    southerntofu, if you think it's easy, I encourage you to try it. It's a lot of work, and complicated.

  38. Zash

    `echo echo BROKEN > compliance-checker.sh`

  39. jonas’

    southerntofu, so, markdown isn't really a format used for XMPP specs to start with ...

  40. MattJ

    The Editor is speaking

  41. southerntofu

    jonas’, could be XML for all i care, i personally like markdown :)

  42. jonas’

    I personally too like markdown.

  43. MattJ

    Meanwhile everyone is using Markdown

  44. jonas’

    one day we should migrate XEPs to use markdown

  45. jonas’

    it would make a lot of things easier.

  46. jonas’

    and more accessible.

  47. jonas’

    it would also make other things harder.

  48. jonas’


  49. jonas’

    different sujbect

  50. jonas’

    southerntofu, so, actually, I really like your idea

  51. Zash

    XMPP flavored markdown

  52. jonas’

    the doctest-style tests in XMPP specs.

  53. Zash


  54. jonas’

    it would be interesting to see something like that for a simple spec like '30 and a non-trivial spec like '45

  55. moparisthebest

    yea they should be written in XEP-0393 instead

  56. jonas’

    /kickban moparisthebest

  57. jonas’

    servers are easy to automate

  58. jonas’

    clients... not so much

  59. southerntofu

    jonas’, i mention multiple (de)serialization formats in "# Other spectest-compliant formats" on that draft blogpost

  60. jonas’

    southerntofu, as I said ... "too long and badly typeset, didn't read."

  61. southerntofu

    XEP-compliant XML could be one of those

  62. Zash

    what jonas’ said

  63. Zash

    It's Friday, not Readlongmarkdownonbacklitscreenday

  64. jonas’

    but in the end, the format is irrelevant

  65. jonas’

    the hard part is actually running the tests

  66. jonas’

    against servers it's doable

  67. jonas’

    but clients... no clue

  68. Zash

    e.g. see scansion

  69. southerntofu

    Zash, TLDR

  70. jonas’

    exactly, scansion

  71. southerntofu

    > As we've seen in the previous articles, open standards and associated test suites are key to achieving expected results, and therefore benefit the whole ecosystem. But often, these two concerns are treated as separate problems to deal with in entirely different ways. As a result, some specifications are so long and complex that coming up with a test suite is a challenge (XMPP/ActivityPub), while other systems are backed by a collection of unspecified test suites that are hard to comprehend (Ansible).

  72. jonas’

    or aioxmpp test suite

  73. southerntofu

    > Could we get the best of both worlds by treating specification and compliance (testing) as a single problem? This hypothetical approach i call specification-driven development, whereby a specification document is intended both for human and machine consumption. In that case, the specification contains a written presentation of concepts, in addition to a machine-readable test suite that follows a certain format to programmatically ensure that the concepts and behavior described in the specification are implemented properly. This format for specifications is called a spectest document. (TODO: maybe specdoc ?)

  74. jonas’

    needs more paragraphs

  75. moparisthebest

    if you can write a specification such that a program can create tests out of it, then a program could also just implement it correctly for you instead ?

  76. moparisthebest

    no programmer needed

  77. jonas’

    moparisthebest, see also IDL

  78. southerntofu

    moparisthebest, the tests would be handcrafted *as part of the spec*

  79. Zash

    Accept: application/epub+zip delivered onto my e-ink device (it has wifi disabled). Good luck!

  80. jonas’

    southerntofu, any practical idea on how to test clients though?

  81. moparisthebest

    I don't actually think that helps in 99% of cases southerntofu

  82. jonas’

    moparisthebest, I think it does.

  83. jonas’

    in many ways

  84. southerntofu

    jonas’, i imagine a client could be intrsumented by running in a "test mode" where it exposes a well-known API (over a socket or something) instructing it to react in a certain manner?

  85. jonas’

    imagine all examples in a XEP could actually be run against servers.

  86. moparisthebest

    the specs are usually clear, and it's easy to test the things in the XEP

  87. jonas’

    southerntofu, ideally without changing the code of existing clients

  88. moparisthebest

    that doesn't help with the 99999 things that actually happen in practice

  89. jonas’

    moparisthebest, I disagree.

  90. Zash

    `gajim-control` you say?

  91. Wojtek

    > a tool to check any server/client for correct implementation would be great uhm at Tigase we have https://github.com/tigase/tigase-tts-ng (and older https://github.com/tigase/tigase-testsuite) which do just that; probably could be nice to generalise it and make available as a service...

  92. moparisthebest

    don't get me wrong, it'd be very helpful, a great start, but you'd still miss a lot of things

  93. Wojtek

    I know :-)

  94. southerntofu

    moparisthebest, if each XEP contained machine tests in addition to human formats, it would help for two things: it would help to clear ambiguities before reaching implementation stage (because if the test contradits human text then somebody will notice earlier and raise issues) and it would help somewhat-correct implementation because you could just do ./xeptest XEP-0030.xml -- myclient --test-mode

  95. jonas’

    moparisthebest, of course, it will never be perfect, but perfect is the enemy of good.

  96. southerntofu

    Zash, i'm not aware of gajim-control

  97. southerntofu

    jonas’, without changing client code i don't think is possible? would be much harder to emulate user interactions on every possible client :) :)

  98. moparisthebest

    we already have "examples are not the spec" surely "tests are not the spec" would be that way too, so I don't think it helps with ambiguities

  99. jonas’

    moparisthebest, but tests would actually be normative

  100. southerntofu

    but maybe the test protocol can be simple enough for all clients to implement?

  101. southerntofu

    jonas’, yes exactly, no ambiguity in there :)

  102. jonas’

    southerntofu, I have no clue how that protocol would look like

  103. jonas’

    except if you accidentally XMPP the protocol.

  104. jonas’

    but that's kind of not the goal

  105. jonas’

    like, if you instruct a client to add a roster item .... and such ... you end up respecifying XMPP

  106. jonas’

    to instruct a client to do XMPP

  107. southerntofu

    yeah but on a higher-level i guess

  108. jonas’

    maybe, but only slightly

  109. jonas’

    it needs to be pretty fine grained if you want to do things like testing individual MUC interactions

  110. southerntofu

    and in fact maybe it can serve as guidelines for library UX? so that if the specs are well-written you could just follow the test API and that would be your client library?

  111. moparisthebest

    I'm not even aware of tools to do non-browser UI testing

  112. jonas’

    ("grant voice to user", "kick user", "unban user"...)

  113. jonas’

    moparisthebest, they exist

  114. jonas’

    they typically use a11y APIs

  115. moparisthebest

    I'm sure they do, just never had the opportunity I guess :)

  116. southerntofu

    i mean i guess all clients/libraries have a "mute" or "ban" function, maybe standardizing that could be part of the answer?

  117. southerntofu

    then it's just a matter of serializing/deserializing instructions to that "API"

  118. jonas’

    southerntofu, I don't see where that could be standardized really

  119. jonas’

    especially across languages.

  120. jonas’

    and UI paradigms

  121. jonas’

    but that standardisation ... it would be like XMPP itself.

  122. southerntofu

    then we could have per-XEP testing interface where we expect you to expose some glue to your client code for specific functions?

  123. moparisthebest

    what are you actually after testing though

  124. jonas’

    southerntofu, same thing really

  125. moparisthebest

    the client author doesn't want to implement a whole other xmpp protocol to see if his "join muc" works, he wants to click "join muc" and see if that works

  126. southerntofu

    jonas’, well per-spec is much easier to implement/maintain/test

  127. jonas’

    southerntofu, doesn't solve the core issue though

  128. southerntofu

    and like i said if you're developing a client/library from scratch, it can serve as implementation guideline for making a good library API

  129. jonas’

    it is another XMPP-ish thing besides XMPP

  130. southerntofu


  131. jonas’

    for libraries, the story is slightly different and we are slowly venturing toward the lands of IDLs

  132. southerntofu


  133. jonas’

    interface description languages

  134. jonas’

    ("interface" as in API)

  135. southerntofu

    ah cool that's sort of what i was talking about :P

  136. jonas’

    like OpenGL publishing an XML file where the entire API is contained so that you can auto-generate headers for any language

  137. jonas’

    but that approach only really works (a) in one single lagnuage/paradigm or (b) for really low-level things

  138. southerntofu

    yeah it wouldn't work for UI testing most likely, but personally i'm fine with that

  139. moparisthebest

    then you just have library testing I guess

  140. southerntofu

    if we reach server & library testing that'd already be a huge milestone :)

  141. southerntofu

    anyway we'll see, when i find a complete week to hack on something and see where it goes

  142. jonas’

    southerntofu, feel free to loop me (the XEP Editor) in.

  143. southerntofu

    can scansion support multiple clients on different servers to keep S2S?

  144. southerntofu

    to test s2S*

  145. Zash

    I know of no reason why it couldn't

  146. southerntofu

    but does it already?

  147. southerntofu

    like can i run two scansion instances with the same "scansion script" where one is client A and one is cliebt B and they're aware of each other's address? and the test can be run on two accounts on the same server (simple e2e test) or across servers (account for s2s bugs)?

  148. southerntofu

    fro my reading of the homepage it does but i'd like to be sure: https://matthewwild.co.uk/projects/scansion/

  149. Sam

    Tried to catch up on the context, but for my library I wrote https://pkg.go.dev/mellium.im/xmpp/internal/integration which is a framework for spinning up servers and clients and linking them up to test against. Works very well for my own integration tests (except for ejabberd which never shuts down cleanly for some reason, so right now it's only running tests that require a server against prosody) and could in theory be adapted to work for other languages and the like and be more stand alone.

  150. Zash

    southerntofu, one scansion script can describe multiple clients, which could likely connect to different servers, but that's not something I've tested

  151. Zash

    Many of the ones we use to test Prosody involve multiple actors interacting

  152. southerntofu

    ok i'll make sure to test that then :)

  153. Zash

    Lots of examples in https://hg.prosody.im/trunk/file/tip/spec/scansion/

  154. Zash

    I imagine changing `jid: juliet@localhost` to @example.com etc would work.

  155. southerntofu

    yeah that was the gist of my question :)

  156. southerntofu

    so it looks like there's at least three different testing frameworks for XMPP lcients/servers (tigase/mellium/scansion)

  157. Zash

    Time to make a testing framework compliance tester!

  158. southerntofu

    would be interesting to see how they compare and whether they could be reunited as part of a single spec

  159. southerntofu

    Zash, maybe not that far haha :P

  160. southerntofu

    i mean tigase and mellium appear (from a quick look) to be both object oriented testing APIs.. if they could have the same interface it *could* be easier to maintain a single testing library with bindings to a bunch of supported languages)

  161. Zash

    Write a scansion script parser in Go and Java?

  162. southerntofu

    yeah wy not, or in any language with C FFI, or..

  163. southerntofu

    i don't know what the best answer is, but i know there's possibilities to explore

  164. MattJ

    The problem is that most libraries try to abstract the protocol, which is different to testing the protocol. You'll only test that the library author's model and assumptions hold true.

  165. MattJ

    That's why Scansion is low-level (but easy), but also tests are typically server-dependent.

  166. southerntofu

    why would test be server-dependent? because the server would return more information if it supports further extensions?

  167. southerntofu

    or because the fields in a stanza would not be serialized in the same order?

  168. MattJ

    Scansion ignores additional elements (in unexpected namespaces) by default

  169. MattJ

    Also element order in most cases iirc (there are some different checking modes)

  170. MattJ

    But there are multiple ways to implement something and be compliant with the spec in many places

  171. MattJ

    Every time you read "MAY" in a spec, that's a gain for implementers and a pain for testers :)

  172. southerntofu

    MattJ, surely there's a *finite* number of ways to comply with the spec, right? if so "OR Juliet receives" (boolean logic) could do the trick?

  173. Zash

    You could copy the script and have one for each possibility. The challenge is not collapsing into a black hole.

  174. MattJ

    The finite possibilities quickly add up

  175. MattJ

    I'm not saying it's impossible, I'm just saying it's a lot of work

  176. southerntofu

    (although like you said scansion as it is would be ill-suited to test clients/libraries, maybe a unified test API would be more suited in that case)

  177. MattJ

    I'm also not saying it's not valuable work, but nobody is motivated to do it after they've already implemented (to their knowledge) a correct XMPP implementation

  178. southerntofu

    MattJ, i do understand the possibility of that in an extensible protoco but in regards to XMPP i don't see, do you have any practical example that comes to mind?

  179. southerntofu

    i've found so far that specs were rather complete or ambiguous but never offered infinite interpretations

  180. MattJ

    I'm not claiming any offer infinite interpretations

  181. southerntofu

    (contrary to ActivityPub/Microformats for instance)

  182. MattJ

    (or that none do)

  183. MattJ

    I encourage you to just try it if you're interested in solving this problem

  184. southerntofu

    sure, i'll certainly try with scansion to test multiple servers

  185. southerntofu

    in the meantime i'd be curious how Wojtek and Sam see each other's testing framework and whether they could imagine to cooperate in a hypothetical future on a single testing API/suite

  186. moparisthebest

    It might be better to have different testing implementations

  187. MattJ

    I think it's more valuable to have multiple, especially if real implementations are built on the same libraries

  188. southerntofu

    well then cooperate on a test format that could be implemented by both testing systems?

  189. MattJ

    A test output format would be nice, to say "I tested XEP-nnnn support, it passed"

  190. Zash

    Aren't there a pile of such formats?

  191. goffi

    Hey there, I've just been hit by https://docs.python.org/3/whatsnew/3.9.html#urllib-parse (Python 3.9 URL parsing does not recognise ";" anymore as a separator, following W3C changes). As we use ";" as a separator in XMPP URIs, that smells like a more general issue for us

  192. Ge0rG

    I've had to work around the Android Java URI parser not recognizing that some years ago already.

  193. goffi

    python has a new parameter, so the fix should be easy there

  194. Zash

    Mmmm, URI vs URL fun.

  195. moparisthebest

    More like URFd

  196. emus

    Ge0rG, southerntofu, Zash, wurstsalat: but we host the xmpp providers list, which tries to reference each entry