jdev - 2025-01-17


  1. jjj333_p (any pronouns)

    how insane would it be for flexibility's sake to roll my own xmpp library, just me, the golang tcp stack, and an xml parsing library

  2. singpolyma

    For any reason other than learning I would strongly suggest to not since several already exist. But it wouldn't be insane

  3. jjj333_p (any pronouns)

    mellium is what ive been looking at so far, but i mainly want the ability to read and send arbitrary xml data in stanzas. this may be possible, im just chewing on ideas

  4. singpolyma

    yeah mellium is what I have used before. I know there are one or two others

  5. jjj333_p (any pronouns)

    do you know if it allows for that?

  6. jjj333_p (any pronouns)

    and yeah theres go-xmpp (more bot oriented, more simple less flexible) and go-sendxmpp which is a tool not a lib

  7. singpolyma

    You can add arbitrary XML payloads for sure

    šŸ‘ 1
  8. moparisthebest

    > how insane would it be for flexibility's sake to roll my own xmpp library, just me, the golang tcp stack, and an xml parsing library Literally everyone does it, it's easy

    šŸ‘ 1
  9. jjj333_p (any pronouns)

    alr

  10. Zash

    Only to give up halfway trough implementing an xml parser from scratch and go on to loudly point out how bad XMPP is?

  11. doge

    "Parsing" xml even when you already have an xml parser is trickier than it seems, by "parsing" I mean building the insane incomplete DOMs which no general purpose xml library supports. You have to keep track of namespaces, namespace prefixes. For example, You can have expat readily convert ns prefixes to namespaces for you, but then you will not be able to see the actual prefix names, which are actually needed for namespaced attributes...

  12. jonasā€™

    what?

  13. jonasā€™

    doge, can you elaborate?

  14. jonasā€™

    I don't see how you need to track namespace prefixes on the application layer to process namespaced attributes (source: I have written a parser which does not expose prefixes to the application and fully supports namespaced attributes)

  15. Zash

    Prosody doesn't and will either drop namespace prefixes completely or turn them into ns1:foo="" etc. XMPP hasn't collapsed, therefore I conclude nothing really depends on namsepace prefixes.

  16. jonasā€™

    *nod*

  17. doge

    That is what I did in my parser too, but I'm just saying that it's an example of absolutely pointless complexity.

  18. doge

    I am not too familiar with XML, I never worked with it outside of xmpp, but reading this https://stackoverflow.com/questions/3931817/xpath-expression-from-xml-with-namespace-prefix/8181480#8181480 I got the impression that xpath is sensitive to these prefixes. Luckily, as far as I know, XMPP libraries do not use XPath anyway because they just come up with their own ad hoc partial dom implementations

  19. jonasā€™

    the xpath _syntax_ is indeed not great for that

  20. jonasā€™

    the semantics are not sensitive to prefixes. there are other xpath-ish things which use a different syntax allowing you to specify the namespace URI instead of the prefix.

  21. doge

    Alright, I just remembered that initially I wanted to use an existing dom library but didn't because of the mess prefixes and namespaces created. Turns out to select ns'd node, you have to also register your local prefixes...

  22. doge

    ah right, I just remembered that initially I wanted to use an existing dom library but didn't because of the mess prefixes and namespaces created. Turns out to select ns'd node, you have to also register your local prefixes...

  23. Zash

    I find I haven't really needed to care much about namespace prefixes, so I don't see the complexity in that. Good demonstration of why you should just use a library tho, or you end up talking like this, often loudly.

  24. doge

    ah right, I just remembered that initially I wanted to use an existing dom library but didn't partially because of the mess prefixes and namespaces created. Turns out to select ns'd node, you have to also register your local prefixes...

  25. doge

    Well, if you are building your own parse, you will need to encounter this. There was actually, I think, only one instance where I saw an attribute prefix in the wild, and it made my initial parser implementation crash.

  26. doge

    > or you end up talking like this, often loudly.

  27. Zash

    Web people don't write their own JSON or HTTP parsers, or they'd complain as much about those :)

  28. doge

    https://draugr.de/upload/00f530bb49b4e1685114d28f8bb6fb44b2fcbc2f/r5dyhkxKI6qSGSkUuf6fhle87sKijlOgHIdpUMyl/zb2rhfKs4RBkthZMMj75v2DJw6knXYFq5B3rCGyqCwevNjScd.png

  29. doge

    > Web people don't write their own JSON or HTTP parsers, or they'd complain as much about those :) Actually you wouldn't need to write an XML parser either if XMPP made use of XML in a remotely sane way.

  30. Zash

    Now, let me tell you about all the quirks we've had with our HTTP and JSON parsers in Prosody! ;)

  31. Zash

    > Actually you wouldn't need to write an XML parser either if XMPP made use of XML in a remotely sane way. I think it's perfectly sane.

  32. jonasā€™

    you need a SAX parser, sure

  33. Zash

    or better yet, an XMPP library that handles all that for you so you can build your app instead of fiddle with low level protocol fiddlery

  34. jonasā€™

    or that, yes.

  35. kalkin

    > or better yet, an XMPP library that handles all that for you so you can build your app instead of fiddle with low level protocol fiddlery I think I have a dejavu. Just a week ago I told doge he doesn't want to write his own library. Moparisthebest said he shouldn't listen to me, this is fine šŸ˜‚

  36. doge

    I had already made a parser by then lol

  37. doge

    Btw it's not like the existing libraries are terribly well documented...

  38. Zash

    kalkin, did moparisthebest semi-jokingly point out that basically every popular client maintains their own library? I don't think that's a good argument to start from scratch.

  39. doge

    Like for example if we take not even a full xmpp library but an omemo library. The only one I could find that could be said to be documented at all would be an async python one which has its own complications. Actually, I think that one has quite thorough documentation, but I mean What few C libraries I coukd find have basically no documentation at all.

  40. doge

    > kalkin, did moparisthebest semi-jokingly point out that basically every popular client maintains their own library? I don't think that's a good argument to start from scratch. Maybe that is actually a good argument because there's gotta be a reason why they do it from scratch. Repeating after others who have already succeeded is often a good strategy.

  41. kalkin

    doge that's an interesting interpretation

  42. kalkin

    Any Dev I talked to, said it sucks that they don't use some 3rd party li

  43. kalkin

    Any Dev I talked to, said it sucks that they don't use some 3rd party lib

  44. kalkin

    > kalkin, did moparisthebest semi-jokingly point out that basically every popular client maintains their own library? I don't think that's a good argument to start from scratch. I think this was implied. šŸ˜

  45. Zash

    I would imagine it's more often of necessity, for lack of a suitable library. Or they ended up forking or maintaining an existing one

  46. doge

    For one, libraries tend to be language specific. It's not as bad if it's a C library, It tends to be easy to port it and to ffi it from a higher level language. And even then, it usually won't be idiomatic for your language. worse yet you have to tie it into your event loop or spawn another thread, and both have complications of their own. That's not to mention that I don't think there are well-maintained c libraries. There is python, but you now have to carry around and embed a python interpreter. You have to code python to glue. At that point you might as well switch your entire project to python, and maybe you don't want to do it that much.

  47. doge

    In the case of dino where they opted to use vala, I don't think reusing any library other than C would've been feasible, and C libraries don't really exist afaik

  48. doge

    And also let's not forget people just like tinkee and they tend to like to tinker in their own language and in their own code. Since nobody is really paying for a new client, the development process has to be fun first and foremost.

  49. jonasā€™

    "C libraries don't really exist" libstrophe, libcouplet, and purple.

  50. jonasā€™

    to name three off the top of my head, I assume there exists at least one more (just because)

  51. doge

    > "C libraries don't really exist" libstrophe, libcouplet, and purple. Last time I checked pidgin barely even supported group chats as in no bookmarks, didn't support omemo, didn't support mam.. As for the other two, I don't know much about them, but how well documented are they exactly? Will it be easier to dive into their code than to just go from scratch?

  52. doge

    > to name three off the top of my head, I assume there exists at least one more (just because) It's not quantity that matters.

  53. jonasā€™

    one of these (not libpurple) is used by profanity (a TUI client)

  54. jonasā€™

    I used one of these (not sure which, it's been almost a decade ago) and it was easy enough.

  55. kalkin

    > "C libraries don't really exist" libstrophe, libcouplet, and purple. Nice

  56. kalkin

    Common Lisp Xmpp libraries don't really exists. jonasā€™: your entry šŸ˜

  57. kalkin

    I actually searched for them and found onl clxmpp which is old...

  58. jonasā€™

    kalkin, nice try, but I don't know the CL ecosystem almost at all.

  59. Kev

    cl-ngxmpp? :)

    ā¤ļø 1
  60. kalkin

    Nice but sadly still missing a lot

  61. rburns

    is it good practice that a bot should ping the server periodically to maintain it's connection? or should that not be necessary?

  62. jonasā€™

    rburns, it is good practice to use XEP-0198 <r/> elements to ascertain that a connection is still live. you shouldn't ping unless the stream has been silent for some time.

  63. rburns

    ok, thanks. looking at the xep. is it right that the <r/> elements should be sent after sending a message.

  64. jonasā€™

    it is one way to do it

  65. jonasā€™

    you can send them whenever though

  66. jonasā€™

    and when I said "ping", I meant "send a <r/> for only liveness-checking-purposes".

  67. jonasā€™

    (there are other reasons to send a "<r/>". there are also other ways to "ping", but <r/> is by far the most efficient)

  68. jonasā€™

    rburns, but, your XMPP library should handle all that, does it not?

  69. rburns

    yeah, looking at the debug log from the xmpp library. (xmpp-client on nodejs) it initially sends the <enable /> element and I can see it respond to <r /> elements with <a/>. nonetheless. the connection is lost consistently after a short time. when the connection is lost. there's no close/disconnect events fired. it silently disappears. I can see the disconnect in the server logs

  70. jonasā€™

    okay, that sounds like a library bug

  71. jonasā€™

    if it does '198, it should either handle reconnection transparently, or it should at least let you know about stuff when it breaks.

  72. rburns

    ok, thanks for your help. will dig around some more.

  73. lovetox

    Are there any known conditions why a server would not put messages to self into MAM ?

  74. Stefan

    jonasā€™: profanity is not using libpurple.

  75. Stefan

    Sorry, did not read correct.

  76. Stefan

    Anyway, Chatty is using libpurple and profanity (and xmppc) are using libstrophe. I think libstrophe is fine. Documentation is ok and it's working.

  77. jonasā€™

    Stefan, glad to see I know my stuff :)

  78. jonasā€™

    (in the sense that i have a rough overview of what happens)

  79. singpolyma

    >> "C libraries don't really exist" libstrophe, libcouplet, and purple. > Last time I checked pidgin barely even supported group chats as in no bookmarks, didn't support omemo, didn't support mam.. As for the other two, I don't know much about them, but how well documented are they exactly? Will it be easier to dive into their code than to just go from scratch? Your library doesn't need to support high level things like bookmarks and mam to be useful. Just doing DNS, connection, TLS, sasl stuff is very time saving by itself. If you want a high level library I agree there are few of those (why I'm making one) but for low level there are lots of fine choices

  80. jonasā€™

    tls, sasl, and I'd add stream management and reconnection logic to it

  81. singpolyma

    > yeah, looking at the debug log from the xmpp library. (xmpp-client on nodejs) it initially sends the <enable /> element and I can see it respond to <r /> elements with <a/>. nonetheless. the connection is lost consistently after a short time. when the connection is lost. there's no close/disconnect events fired. it silently disappears. I can see the disconnect in the server logs rburns: is this for xmpp.js ? We are adding proper handling of this in the main dev branch just in the past weeks. Should be in next release

  82. jonasā€™

    you really don't want to bolt that onto a library as an afterthought, it doesn't end well.

  83. singpolyma

    I mostly agree yes

  84. doge

    > Just doing DNS, connection, TLS, sasl stuff is very time saving by itself I don't think implementing those yourself is more hassle than having to deal with ffi

  85. singpolyma

    What stack are you in?

  86. rburns

    singpolyma: yes, after more digging I found your PR recently merged which looked related. Look forward to the next release

  87. doge

    tcl

  88. singpolyma

    I thought calling C from TCL was something it is good at?

  89. doge

    Writing a c extension is indeed rather easy. But with xmpp you deal with a dom. You deal with a recursive structure, you have to tie that in with whatever high language object system you're using and its garbage collection. With Tcl it could be even easier to do that than perhaps with python, but wouldn't be tcl-esque at all or convenient to use. Connecting it well enough for it to be pleasant to use would be at least as much work as just re-implementing the DOM at tcl level imho

  90. doge

    And then you also have to integrate it with the event loop. Which is easier but still another obstacle. So these obstacles combined I think make it easier to just reimplement it in tickle.

  91. moparisthebest

    Huh? What's recursive ?

  92. doge

    dom nodes

  93. moparisthebest

    I mean a dom is a tree, but so are like your favorite json nodes šŸ¤·ā€ā™‚ļø

  94. doge

    Yes, but for json you already have a language specific library which also isn't insane with namespaces prefixes etc etc. Most XML libraries seem so huge and bloated and have so many features that honestly it may be for the better to just not use them in a networked application....

  95. singpolyma

    Yeah you only care about prefixes at parse time. You don't need them in the data model

  96. doge

    If there was a library that handled authentication, stream management and reconnecting, and then gave you like just an XML stream that you would reparse, or better yet provide a length prefix stream of normal XML documents you could pass into an existing xml library, perhaps I would use that.

  97. singpolyma

    Sounds like xmpp-proxy

  98. singpolyma

    Just using a sax parser is easier imo but it does exist

  99. doge

    > Yeah you only care about prefixes at parse time. You don't need them in the data model I meant the other kind of prefixes. It turned out that to use namespaced nodes from an XML library you actually need to register your own local prefixes, and to insert those at every corner..

  100. doge

    Not as is a big problem, but it just highlights the level of insanity we're dealing with and the tendency to create problems out of nowhere.

  101. singpolyma

    Sure some libraries do thinks that way because people prefer that syntax to clark notation or whatever

  102. singpolyma

    Sure some libraries do things that way because people prefer that syntax to clark notation or whatever

  103. doge

    I honestly prefer to just ignore the namespaces for most of the time. Or "same namespace as the parent until otherwise specified"

  104. singpolyma

    I've seen some libraries to that. I find it a terrifying way to operate, but if you're careful it can work

  105. moparisthebest

    > Yes, but for json you already have a language specific library which also isn't insane with namespaces prefixes etc etc. Most XML libraries seem so huge and bloated and have so many features that honestly it may be for the better to just not use them in a networked application.... doge: https://www.moparisthebest.com/images/jsonschema.jpg

  106. Zash

    moparisthebest, don't forget JSON-LD and its namespace prefixes!

  107. moparisthebest

    > moparisthebest, don't forget JSON-LD and its namespace prefixes! Indeed... In the end to implement something sufficiently extensible for XMPP you'd need all this "complexity" in json too, which ends up being even worse imho

  108. jonasā€™

    and its namespace prefixes WHICH ARE USED IN VALUES

  109. jonasā€™

    they're making the same mistakes as SOAP again.

  110. jonasā€™

    making it absolutely impossible to decouple parsing and application

  111. doge

    >> Yes, but for json you already have a language specific library which also isn't insane with namespaces prefixes etc etc. Most XML libraries seem so huge and bloated and have so many features that honestly it may be for the better to just not use them in a networked application.... > doge: https://www.moparisthebest.com/images/jsonschema.jpg "Look, somebody made this stupid thing on top of this sane thing. It means our stupid thing is not as stupid after all! "

  112. moparisthebest

    doge: s/stupid/required/ but sure

  113. doge

    > they're making the same mistakes as SOAP again. SOAP's mistake was existing.

  114. jonasā€™

    indeed.

  115. singpolyma

    I don't mind using soap with a good library but I'd never want to implement it

  116. singpolyma

    SOAP is back now anyway they just call it graphql now. What's old is always and ever new again

  117. singpolyma

    This has all happened before and it will all happen again

  118. jonasā€™

    what is dead may never die

  119. Kev

    This is the way.

  120. Zash

    So say we all

  121. jonasā€™

    It is known.

  122. moparisthebest

    It does seem like many software developers are opposed to using soap...