jdev - 2024-07-04

  1. moparisthebest

    taba: https://wiki.archlinux.org/title/Tor#Transparent_Torification

  2. taba

    > maybe you should listen to what people suggest before just asserting that they're wrong qy: i know what he's talking about toxigoosy πŸ™„

  3. taba

    > taba: https://wiki.archlinux.org/title/Tor#Transparent_Torification > In some cases it is more secure and often easier to transparently torify _an entire system_ instead of configuring individual applications to use Tor's socks port, not to mention preventing DNS leaks.

  4. taba

    > taba: https://wiki.archlinux.org/title/Tor#Transparent_Torification > In some cases it is more secure and often easier to transparently torify *an entire system* instead of configuring individual applications to use Tor's socks port, not to mention preventing DNS leaks.

  5. qy

    > not how tor works... you need clients designed with tor in mind it didnt sound like it, but sure

  6. taba

    > Transparent torification also will not protect against fingerprinting attacks on its own

  7. taba

    >> not how tor works... you need clients designed with tor in mind > it didnt sound like it, but sure https://support.torproject.org/#faq_staying-anonymous πŸ™„πŸ™„πŸ™„πŸ™„πŸ™„πŸ™„πŸ™„πŸ™„πŸ™„πŸ™„πŸ™„πŸ™„πŸ™„πŸ™„πŸ™„πŸ™„

  8. qy

    "not how tor works"

    πŸ™„ 1
  9. taba

    > "not how tor works" πŸ™„

  10. wgreenhouse

    the firewall-rule thing can also be done to one user, or a network namespace (cf. orjail)

  11. taba

    πŸ˜”the module doesn't seem to work anywhere

  12. MSavoritias (fae,ve)

    does xml care if its tab or spaces? right now i am thinking in context of the doap but probably its applicable everywhere

  13. MSavoritias (fae,ve)

    the default in gnome builder seems to be spaces

  14. Link Mauve

    MSavoritias (fae,ve), you can use either, and in most cases they are irrelevant.

  15. MSavoritias (fae,ve)

    actually i have a better question: is there something i am missing about defining a XML Schema Definition Language (XSD) for XMPP and then use that to restrict what we can receive? probably we are losing much of the extensibility right?

  16. MSavoritias (fae,ve)

    > MSavoritias (fae,ve), you can use either, and in most cases they are irrelevant. ah ok. thats good

  17. Link Mauve

    xml:space controls some of that.

  18. MSavoritias (fae,ve)

    so xml is basically a lisp then :P

  19. Link Mauve

    MSavoritias (fae,ve), in XMPP we usually don’t use the XML Schema for much, I’ve heard of some server software using them to validate client input, but I’ve never used that.

  20. Kev

    There are definitely organisations that do validation of one form or another at their boundaries as a security thing.

  21. Link Mauve

    There is also EXI which is meant to negociate that between the client and the server.

  22. MSavoritias (fae,ve)

    i thought EXI was only a compression binary thing. but i guess since it does specify what exactly the things inside are it can be for the same thing

  23. MSavoritias (fae,ve)

    > There are definitely organisations that do validation of one form or another at their boundaries as a security thing. yeah that was my thinking. it would make parsing easier if i knew what to expect

  24. Link Mauve

    I tend to use the schemas as a guideline about what to accept in my software, more as documentation.

  25. jonas’

    MSavoritias (fae,ve), you don't want to konw what XML does to whitespace :-)

  26. MSavoritias (fae,ve)


  27. jonas’

    in particular: within attribute values, tabs, spaces, carriage returns and newlines are folded to space. outside of attribute values, CRLF is converted to LF

  28. jonas’

    and this is non-optional XML 1.0 behaviour.

  29. MSavoritias (fae,ve)

    ah okay. i was referring to outside of that

  30. MSavoritias (fae,ve)

    but noted not to touch anything inside the xml :)

  31. Guus

    Is there a server developer (specifically someone who works on an XMPP server project that has a build pipeline, for example in GitLab, GitHub, Bamboo or CircleCI) that is interested in helping Fishbowler and me with a new XMPP interop testing project that we've been working on these past few months?

  32. Guus

    We're looking for someone with fresh eyes, an alpha tester if you like. We've been so heads-down in this project, that we might miss things that we incorrectly assume are obvious. We'd like someone else to verify / comment on the usability of our project, so that we can improve things.

  33. Guus

    Depending on your pre-existing CI implementation, I expect this to take somewhere between one to four hours.

  34. MattJ

    We self-host our CI in buildbot

  35. Guus

    MattJ, I think that won't be compatible at this stage. I'm trying to dive into its documentation (much of it 404's) - I think it supports a plugin architecture, which we might be able to leverage in the future to add direct support.

  36. MattJ

    Why does it need special support from the CI system?

  37. MattJ

    and what 404s?

  38. Guus

    The project's main purpose is to generate specific

  39. Guus

    plugins/orbs/actions that can natively be added to the CI system that's being used.

  40. Guus

    we've simply not created something for buildbot yet.

  41. MattJ

    Seems more restrictive than just producing a standalone tool that can be used in any environment

  42. Guus

    many links found on https://buildbot.net/ (under 'get started' for example) yield 404 responses.

  43. MattJ

    Sometimes I might want to run tests outside of CI

  44. MattJ

    The Buildbot docs are at https://docs.buildbot.net/4.0.0/manual/index.html

  45. MattJ

    Not sure what's going on with the site, I'll let them know

  46. Zash

    Seems they link to /current/ but readthedocs does /latest/

  47. Guus

    I believe that the cake is mostly a lie, with regards to being able to create a stand-alone tool. If anything one would exist by now. What we produce does use a common component that's wrapped in CI-specific builders, which you can run as such a stand-alone tool. I'm pretty sure that as soon as I disclose its nature, people will start having a debate about its nature / runtime requirements, etc.

  48. flow

    I think MattJ is just saying that an interop tool does not has to be CI specific

  49. MattJ

    If something not existing can be used as evidence that it's not possible, I have news... :)

  50. Guus

    That's the bit I'd like to avoid with providing CI-specific integrations that are easily hooked into existing pipelines. The drawback is that we need specific support for each CI system.

  51. flow

    I am thinking "simply wrap the XMPP servers into containers, wrap a script around it, that starts a server cointainer, points the test suite at it and repeats this for every server implementation"

  52. Guus

    OK, let me put it this way: it _has_ existed (for at least a decade), and no-one except for Openfire was using it.

  53. MattJ

    And now nobody is using it because it's not compatible with our build environment, or any of the future build environments

  54. MattJ

    Sorry, that probably came out a bit harsh, I'm not intending to say you've done a bad job

  55. MattJ

    It just feels like an unusual design decision, that only restricts the applicability of the project

  56. flow

    I am thinking "simply wrap the XMPP servers into containers (or a "local" temp installation), wrap a script around it, that starts a server cointainer, points the test suite at it and repeats this for every server implementation"

  57. MattJ

    Making it work only on a defined list of specific provider APIs seems like it's more prone to becoming obsoleted in the future as the set of popular providers changes, or as existing providers update their APIs and interfaces

  58. Guus

    Before starting, we've tried to analyze what CI systems were in use by server implementations that we could find. I think we've added support, or are planning support for all build systems that we could identify that were in use, with the exception of Prosody's.

  59. MattJ

    Whereas how to execute a standalone executable has not changed for decades

  60. MattJ

    So how many server implementations do you cover?

  61. flow

    I now start to wonder everyone is one the same page. I think you can run guus' tool standalone without a CI and simply point it to any server (implementation)

  62. MattJ

    If that's true, then I take everything back :)

  63. flow

    the result will be a list of tests that passed/failed

  64. MattJ

    But it sounded to me like the whole thing only works in specific CI environments

  65. Guus

    MattJ, I expect that we're currently covering the CI systems used by 7 or 8 of the server projects

  66. flow

    basically the tool us based on Smack's integration tests, but on steriods

  67. Guus

    out of the 9 server projects that I tried to find the CI system in use for.

  68. flow

    basically the tool is based on Smack's integration tests, but on steriods

  69. MattJ

    I'm surprised we have 9 active server projects

  70. Guus

    Astrachat, Openfire, EJabberd, Tigase, Prosody, M-Link, Jackal, Metronome, Mongose

  71. MattJ

    Openfire, ejabberd, Mongoose, Prosody... do you cover closed-source ones?

  72. Guus

    various degrees of 'activeness'...

  73. MattJ

    Oh, Tigase of course

  74. flow

    And Astrachat is a from scratch implementation or a fork of?

  75. MattJ

    IIRC the Jackal maintainer announced it was no longer being developed

  76. Guus

    anyway, I'm not really looking to debate the approach that we've already committed to. I definitely want to somehow be able to service Prosody too - either by a buildbot specific thing, or by something more generic like docker, or just bare-boned Smack SINT .

  77. flow

    Guus, I am not sure if there is anything wrong with the approach, it's just that I find it strange that you talk so much about CI that it sounds like the approach requires a specific CI system

  78. flow

    of course, I could be misunderstanding something

  79. MattJ

    I just don't understand what you could possibly need to do that would require a Buildbot plugin that you can't just do directly

  80. flow

    but basically you/we have a integration test runner and a set of integreation tests that you can point to any XMPP port

  81. flow

    simply by "./run <ip-or-host>"

  82. Guus

    flow, that's because that I came here today looking for someone that can provide feedback of integrating our stuff in their pre-exising CI system

  83. MattJ

    Right, and we have a pre-existing CI system and I have a desire to add more tests and to help you, but I feel excluded for some reason I can't fathom :)

  84. MattJ

    If flow is right, and it's possible to just run the thing as a normal executable, we can of course do it

  85. Guus

    MattJ, in most CI systems, you _can_ do anything, with enough time spent. We've build something to take away much of that effort (hopefully), by providing neat integrations.

  86. flow

    MattJ, if you are able to ramp-up a test prosody instance as part of your CI, then it should be as simple as "git clone && ./run" (of course, I simplify things here)

  87. MattJ

    But if you're doing specific stuff with proprietary CI APIs then no

  88. MattJ

    flow, sure, that's exactly what we do with our existing tests

  89. Zash

    ramping down is the hard part

  90. flow

    I figured, the talk about CI was just red herring

  91. MattJ

    So I was expecting this to be similar

  92. Guus

    The CI integrations are mostly wrappers for common stuff. Take input, install a runtime environment and its requirements, execute tests based on user input, parse results.

  93. Guus

    Zash, we've found that in most CI systems, the ramp down was not needed, as runners would simply be thrown away after execution anyway.

  94. Guus

    We actually ended up removing rampdown at some places, as it could only introduce failure for something that was ... basically pointless.

  95. Guus

    That may not be true for all pipelines, of course

  96. Zash

    That's not how BuildBot works

  97. Zash

    It's pretty backwards from all the modern container based CI in use today

  98. flow

    so buildbot is similar to classic jenkins?

  99. Zash

    I do believe it's exactly Jenkins translated into Python

  100. flow

    I am not a "container rule everything around me" person, but once we switched our CI to a container based one, the advantages became obvious

  101. flow

    that was probably 8 years or so ago

  102. flow

    that was probably 8 years or so ago :D

  103. Guus

    Next on our list is to create various container-based approaches to run these tests - we've simply not gotten there yet.

  104. flow

    I think you only need to consider containerizing the servers, the integration test runner has little need for containerization as its java based (so java/the jvm is your container platform)

  105. flow

    of course, one could also put it into a container and then do containers-in-containers (not kidding, that can be sensible in some cases)

  106. Guus

    we're not touching the servers.

  107. flow

    I think what would be a huge step forward for XMPP interop (testing) is, if every server implementation also provided an official container (test) image

  108. Guus

    The only requirement that we basically have is: make sure that a server is network-reachable (and can be provisioned with users, either through inband or adhoc)

  109. Guus

    I'm not saying it is a bad idea - it is just something that we do not require.

  110. Guus

    We will provide containers with our tests, assuming that there are people out there that prefer to run a standard container format over a standard Java VM

  111. Guus

    containers that contain our test clienit/runners, I mean

  112. Guus

    (it'd also allow us to eventually build additional tests that are not written in Java, without requiring users to re-do their CI setup)

  113. flow

    not written in Java, you must be referring to Scala 3 then :)

  114. Guus

    we have no immediate plans, but we did consider that it'd be nice to retain the same interface (from the CI's perspective) and be able to cram in whatever we need for test execution. Today, it's all Smack / SINT, but maybe we want to add something like aioxmpp, or something completely different.

  115. Zash

    Guus, MattJ: I opened https://github.com/buildbot/buildbot-website/pull/87

  116. Guus

    but, to re-iterate: we're looking for help from a server developer that (pending support for more environments in the near future) is using a CI system that we currently support, to verify that what we are producing in documentation and runtime feedback/output is ... sensible.

  117. Fishbowler

    ^this^ - feedback on the ease of integration and the quality of the feedback are core goals of the project On Guus' estimate, if it takes closer to 4 hours, rather than closer to 1 hour, we'd really want to know about that.

  118. flow

    I am surpised that there seems to be a strong focus on a particular CI system. The one thing I've learned from Travis CI was to design CI in such a way that the CI system doesn't matter much…

  119. Guus

    There's no focus on a particular CI system at all. We've created tests and docs/tooling for those to be easily integrated in what we believe is ~80% of the existing tools used today (with all intentions to develop integrations for other scenarios in the near future), and are looking for an outsider to validate what we've built so far - again, with clear intentions to continue developing things further.

  120. flow

    ok, the "is using a CI system that we currently support" part then confuses me

  121. flow

    Ideally you would even want someone *not* using the CI system that you have used, to see if there are any challenges

  122. flow

    at least, that is what \me thinks

  123. flow

    probably I am missing something

  124. Guus

    we currently support ... every CI system used by a server project - except for maybe only Prosody at this stage.

  125. Guus

    we're also in no way finished - additional tooling / docs for other CI environments, including the one used by Prosody - will be added soon

  126. flow

    what's the piece that is assumed missing for prosody's CI system?

  127. Guus

    but at this stage, we're just looking for some confirmation that the stuff that we _have_ build is not void of any major oversights. Fishbowler and me have been so neck-deep in this project, that we might be overlooking something obvious.

  128. Guus

    For Prosody, either a containerized version of the test runner or documentation on how to run tests on 'bare metal' will likely be beneficial.

  129. flow

    so it's more missing documentation and not missing technology that makes prosody's CI system "unsupported"

  130. Guus

    possibly - we've simply not gotten around to that. I'm not sure if we end up documenting usage that's completely uncontainerized (as that has some drawbacks too - what is now a black box suddenly becomes part of an API of sorts) - it certainly is not out of the question.

  131. flow

    I totally get that you want someone else to try it, what I don't get is that, even in the presence of prosody folks showing interest in the integration, you seem to have taken a strong position that it isn't ready for prosody

  132. flow

    whereas all they seem need is a jvm to execute the integration test runner

  133. flow

    prosody has IIRC already integration tests, so the infra to ramp up the server is already there

  134. flow

    what's missing is to excute the same thing with a different integration test run

  135. Guus


  136. flow

    what's missing is to excute the same thing with a different integration test runner

  137. flow

    and that should be trivial

  138. Guus

    I'm not saying that we're not going to build that

  139. Guus

    I'm just asking for help with stuff that we've already built.

  140. flow

    ok I am saying you already have build that

  141. Guus

    that, sadly, does not seem to be very applicable to Prosody _het_.

  142. Guus


  143. flow

    that you have already build that seems to be one (major) thing where we have a different understanding

  144. flow

    basically, replace https://hg.prosody.im/trunk/file/tip/GNUmakefile#l114 with sinttest and done

  145. flow

    no ci, no container foo, it's a dream :)

  146. flow

    no ci, no container foo, it's like a dream :)

  147. Guus

    The test implementations is not what I'm looking for today. Apart from writing test implementations, we have compiled a lot of other things (documentation, configuration, examples, stuff that generates platform-specific output) that are very specific to CI environments. Those are the things that I'm today looking for feedback on.

  148. Guus

    Don't get me wrong, I'd also love to hear if our tests run successfully on Prosody - but that will open a completely parallel line of feedback

  149. flow

    ok, I see

  150. Guus

    specifically, one of the questions that I have is if when a test fails, if a developer can figure out why it fails. I'm looking, for example, for confirmation if the very specific way that we've currently configured sinttest to execute within our CI abstractions, combined with our documenation and configuration examples, are sufficient, or need to be improved.

  151. moparisthebest

    > I am thinking "simply wrap the XMPP servers into containers (or a "local" temp installation), wrap a script around it, that starts a server cointainer, points the test suite at it and repeats this for every server implementation" That's exactly what I do here https://github.com/moparisthebest/xmpp-proxy/blob/master/integration/test.sh I'm with MattJ though, make it trivial to run from a terminal then a few examples of integrating into popular at this moment CIs is all that is needed

  152. Guus

    What happens if a test fails?

  153. Guus

    As an example, we've been using aioxmpp for integration tests. It has flagged a couple of interesting bugs, and was very valuable.

  154. Guus

    what was an absolute pain to me is trying to figure out _why_ a test fails

  155. Guus

    first, you have to comb through the output to figure out what test fails, then you have to find the corresponding implementation, try to interpret that (which isn't easy if it's writting in a language that you're not familiar with) and then try and deduce what's going wrong.

  156. Guus

    we've tried to make that all a lot easier, by - hopefully - give you tools to not have to look at the test implementation at all. On the one hand, we've put effort in publishing per-test XMPP stanza output. We've also added references to as specific as possible the specification that each particular test is failing. Also, we've modified each test assertion in such a way that they hopefully 'make sense' when looking at them without looking at the test implementation.

  157. Guus

    that's all hooked together in our test runners - that probably _can_ be made avialable too as a stand-alone runner (but we haven't built that yet).

  158. Guus

    That's what we're asking for help with - if the tooling that we've provided is adequate to consume the output of the executed tests - which maybe is more than just execute them and be done with it.

  159. Guus

    (and, as we've bundled that up in CI-specific runners, I'm looking for someone that uses one of the CIs that we already support)