XMPP Service Operators - 2024-01-27

  1. Sam

    I have an odd very specific request. I need someones expertise and would prefer if they could reply to the question directly, but it's on an internal communication tool, so I'm wondering if there's anyone here who's both run an XMPP server (I hope so since you're in this room), and also a *non* Element Matrix server, and is *also* an IWW member who would want to share some performance or operational comparisons?

  2. Sam

    (TL;DR there has been some discussion of moving the IWW off of the commercial chat provider they're using now and onto something open and likely run in house; very early discussion, but of course the usual arguments for Matrix are being thrown around and I can counter the "XMPP is slow and a bandwidth hog because it uses XML" and what not, but don't have any hard numbers on why I suspect Matrix will *always* use tons more resources even if we avoid the badly written Element stuff which, admittedly, is where most of the performance issues come in)

  3. Martin

    Don't know what's IWW but disroot.org ditched matrix and went back to xmpp due to resource usage.

  4. Bob Evans

    What is the IWW.

  5. Bob Evans

    How many users.

  6. Martin

    The admin muppeth used to hang around in some MUCs but unfortunately he doesn't seem to be here.

  7. MattJ


  8. MattJ

    And this summary from the Matrix trial the IETF ran: https://mailarchive.ietf.org/arch/msg/tools-discuss/bdGVrXm7GxMu704HPIXYkJPH6NQ/ (they went with Zulip, which is open-source but not federated and recently announced they'll start charging self-hosted deployments that use the Zulip apps)

  9. Menel

    Web search for IWW only finds German a GmbH for me ๐Ÿ™‚

  10. Guus

    You'd think that having a discussion on resource usage instead of privacy or functionality is rather pointless / much alike bikeshedding, until you read '91 GB for 37 local users.' Holy crap.

  11. MSavoritias (fae,ve)

    also https://matrix.org/blog/2024/01/migrating-from-ems-to-selfhosted-matrix/

  12. MSavoritias (fae,ve)

    ironicall this says a lot in itself

  13. MSavoritias (fae,ve)

    specifically it needs at least 4-5 hours to catch on messages for one person instance

  14. MSavoritias (fae,ve)

    during a migration

  15. MSavoritias (fae,ve)

    and the server is unsable during that time because of cpu being used at 100% of course

  16. MSavoritias (fae,ve)

    iww is https://www.iww.org/ afaik btw

  17. MSavoritias (fae,ve)

    also of interest https://telegra.ph/why-not-matrix-08-07 it has architectural critisisms of the matrix protocol

  18. moparisthebest

    See also: all the popular matrix channels constantly getting "bricked" and having to be destroyed and remade elsewhere to be used again

  19. MSavoritias (fae,ve)

    right forgot. one example https://github.com/matrix-org/synapse/issues/14481

  20. MSavoritias (fae,ve)

    > Our main 2 Matrix rooms were recently bricked by Matrix bugs in the state resolution protocol which is why they're so small compared to the more niche rooms. The main room had over 15k people before getting bricked in November 2022 and over 13k before getting bricked again last month.

  21. Bob Evans

    15k people doesn't seem believable.

  22. MSavoritias (fae,ve)

    matrix doesnt remove people from a room. so they can pile up over time

  23. MSavoritias (fae,ve)

    thats why it "seems" there are a lot

  24. moparisthebest

    See the Eagle's song "Hotel California" which I assume was their inspiration in designing the protocol

  25. Licaon_Kter

    They need to have a way to brag about _"muh users numberz"_ in every PR opportunity, as you can't sell private hidden invisible protocol users/servers. I guess Guus's server info graphic thing tries to convey _"xmpp exists you guys"_ but in a privacy way

  26. MSavoritias (fae,ve)

    yeah people are so surprised there are some many servers when i show them :D

  27. Menel

    Some people say dendrite is much better, I guess all the examples are from synapse. Thst may be why Sam asked about other servers.

  28. MSavoritias (fae,ve)

    other servers are in beta. there is conduit which is still beta and not recommended. according to their docs: What is the current status? Conduit is Beta, meaning you can join and participate in most Matrix rooms, but not all features are supported and you might run into bugs from time to time. There are still a few important features missing: E2EE emoji comparison over federation (E2EE chat works) Outgoing read receipts, typing, presence over federation (incoming works) Check out the Conduit 1.0 Release Milestone.

  29. Menel

    Or was it conduid? Whatever

  30. MSavoritias (fae,ve)

    and dendrite probably is completely on hold. the CEO of Element said that all projects (p2p, dendrite, whatever) are dropped because there is no money

  31. Sam

    Thanks all, these links are really helpful. Yes, sorry, IWW is the Industrial Workers of the World. There are probably between 12,000 and 15,000 members these days (though of course not all of them will be on the same chat, generally independent branches maintain their own chat stuff, so this is why a federated alternative would be better)

  32. MSavoritias (fae,ve)

    there are some people running conduit on prod but its all single user instances with major caveats. nobody should be running this for prod for many people

  33. Sam

    The reason I asked about non-Element servers is that any time you bring up performance issues they always just say "that's just Element, use dendrite" or whatever. That takes care of a lot of it, to be sure, but in general you still have to do the whole "giant graph of all the messages" thing, so I was hoping to find someone who could respond to the thread directly and had experience (I know it's a long shot though)

  34. MSavoritias (fae,ve)

    i havent heard of anyone running conduit or dendrite for more than a single user or a private setup personally

  35. MSavoritias (fae,ve)

    so that should be enough

  36. moparisthebest

    The bricking of rooms is likely a protocol problem, the resource usage certainly is

  37. Sam

    I didn't know about the bricking of rooms; I'll have to look into it, thanks. It makes sense if you're going to do your protocol that way though. Any issue with propogation of the graph and you're screwed.

  38. Sam

    But meanwhile the usual "XMPP doesn't have any QoS so messages might not be delivered!" is being trotted out.

  39. Sam


  40. MattJ

    "L" "O" "L"

  41. MSavoritias (fae,ve)

    there was a person at one point reading through the matrix protocol and saying that a *lot* of stuff is unspecified in there. but cant find the link

  42. MSavoritias (fae,ve)

    its definetily some fundamental things

  43. Licaon_Kter

    Sam: every time you raise the single server/client from single company/sponsor issue they deflect with _"but dendrite you guys"_ and _"but conduit"_ etc They've done this for the last 6 years or so. Basically when they had the money they kept scaling the python server in all directions and didn't care about dendrite etc, just pumped money in matrix dot org big server and hoping that modular.im oh wa- element.io will start flying... I guess it did not? Now, their slides are really say, 2025 will be brutal....

  44. Sam

    That's probably not a good tract to take given how many times I've been unable to determine what the behavior should be in our own specs, and maybe for me specifically since every time I go back and read my own stuff I find a ton of unspecified stuff that I somehow totally missed :)

  45. MSavoritias (fae,ve)

    fair probably :P

  46. MSavoritias (fae,ve)

    why cant you just deploy both of them and do a trial run? or is that a stupid question

  47. MSavoritias (fae,ve)

    thats kind of what i am doing with guix at some point :)

  48. Sam

    That's a fair point

  49. MSavoritias (fae,ve)

    a vps for xmpp should cost anything either way compared to matrix :P

  50. MSavoritias (fae,ve)

    a vps for xmpp shouldnt cost anything either way compared to matrix :P

  51. Licaon_Kter

    Sam: on xmpp, I bet I can start a client from 2018 and connect to ejabberd 24.01, or a prosody 0.11 from 2018 and connect to the latest client release and it will just work. But you know all that :))

  52. Sam

    Although I worry the trial wouldn't have enough people on it for the issues with Matrix to become aparent, then it would just be picked because the clients are shinier (with apologies to XMPP client developers, but I think it's true that for most people the element ones just look like what they expect a client to look like)

  53. MSavoritias (fae,ve)

    yeah true true :/

  54. Licaon_Kter

    Sad but true, both points :))

  55. Sam

    Licaon_Kter: Is that not true of Matrix? That's definitely a huge thing to bring up if so, thank you!

  56. Licaon_Kter

    No, jokes aside the ever moving _"room version"_ is afaik an issue.

  57. MSavoritias (fae,ve)

    yeah good point

  58. Licaon_Kter

    _monolithic_ argument is nil

  59. MSavoritias (fae,ve)

    i would also add that anything outside of element falls apart

  60. Licaon_Kter

    All I saw was _"update your server/client"_ or else after version X you can't join or server can't join or etc

  61. MSavoritias (fae,ve)

    i was trying the other day to join the kde space with a qt matrix client thats supposedly good. was waiting for it to sync 2 and a half hours

  62. MSavoritias (fae,ve)

    new account

  63. moparisthebest

    Sam: for the trial rent 2 256mb VPS's, one for matrix and one for XMPP >:)

  64. Sam

    hah, yah, good idea

  65. Licaon_Kter

    moparisthebest: don't be like that

  66. Licaon_Kter

    The XMPP one would host _only_ 100 users.

  67. Licaon_Kter


  68. Licaon_Kter

    Or my RPi1/256Mb did that back in 2018

  69. moparisthebest

    Yes, rent those to run the trial on, good idea

  70. Bob Evans

    Can an XMPP server host 15k users.

  71. Bob Evans

    In one channel.

  72. MattJ


  73. MattJ

    Not with default settings, which are generally optimized for smaller channels, but it can

  74. Bob Evans

    What settings have to be changed.

  75. MattJ

    That's up to you and your use deployment's use case

  76. MattJ

    That's up to you and your deployment's use case

  77. Bob Evans

    I was curious. I don't have a use case.

  78. MattJ

    A 15k channel I know of used selective presence via https://modules.prosody.im/mod_muc_batched_probe

  79. MattJ

    Presence broadcasts are one of the most inefficient things at scale (e.g. Matrix supports presence, but it's disabled on matrix.org)

  80. MattJ

    And from a user perspective, you don't need to know the connectivity status of 15k other users

  81. Bob Evans

    Thank you.

  82. Licaon_Kter

    > And from a user perspective, you don't need to know the connectivity status of 15k other users ....and that's why matrix keeps them all "as online" ? Lol

  83. MattJ

    I think that's a UX issue that's (mostly) separate from the protocol. You can do both ways in XMPP, and I think in some cases it makes sense (it's why a lot of people think they want MIX, for example).

  84. MSavoritias (fae,ve)

    i think matrix keeps everybody online due to the protocol designed that way specifically.

  85. MSavoritias (fae,ve)

    dag and all that

  86. Licaon_Kter

    Everytime I see this question the answer is _"it's possible"_ but not *how* Prosody has a module, maybe ejabberd devs speak about MucSub But, is anyone doing it? Where? How?

  87. matthias

    > I think that's a UX issue that's (mostly) separate from the protocol. You can do both ways in XMPP, and I think in some cases it makes sense (it's why a lot of people think they want MIX, for example). Good point. Any news from MIX? ๐Ÿ˜€

  88. MSavoritias (fae,ve)

    2024 is the year of MIX? /s

  89. moparisthebest

    Yes: still seems no one needs it enough to implement and deploy it

  90. MattJ

    You don't need MIX to do this

  91. MattJ

    It doesn't even *need* a module. It's that most clients still choose to use presence instead of membership.

  92. savagepeanut

    Does anybody actually _want_ 15k in IM ?

  93. Menel

    Not active chatting ๐Ÿ™‚

  94. matthias

    I would still love MUCs that does not loose connectivity when a server restarts or when s2s connection was down or when a server changed IP. But perhaps this would not be fixed with MIX ^^

  95. matthias

    I know that prosody did a lot to fix this but as long as there are ejabberd server around..

  96. moparisthebest

    Muc ping or s2s SM fixes that, I don't believe mix actually does

  97. MattJ

    MIX currently makes that worse (there have been proposals for ways to fix it). MIX stores messages in user archives, instead of using a single archive like MUC does. But this means if message delivery fails (e.g. due to s2s issues) then it will be missing from your archive, and there is no way defined to sync or re-fetch such messages.

  98. MattJ

    In MUC you just re-sync the room history with the MUC when you come online

  99. kapad

    ... is disroot xmpp server down ? anyone ?

  100. MattJ

    https://connect.xmpp.net/?disroot.org suggests so

  101. MattJ

    https://status.disroot.org/ says it's up though

  102. Menel

    Can't reach it too. So it seem the monitor tool doesn't monitor correctly in addition

  103. kapad

    ยป https://connect.xmpp.net/?disroot.org , yes seems this work like my clients here, timeout for `s2s`, and ??? for `c2s`. thanks.

  104. Menel

    Up again, but long responding time

  105. chunk

    hi kapad

  106. chunk

    add my jid {me}@xmpp.bot and i can invite you to new subnet place (temp)

  107. chunk

    or, or, powers of deduction, things

  108. kapad

    Menel: yes participants are 170 and goes up, seems like a reboot/restart

  109. chunk

    i see 202 here

  110. kapad

    chunk: ๐Ÿ–๏ธ all, ok?

  111. chunk

    all good my friend

  112. chunk


  113. kapad


  114. klaudie

    Menel: use MTR on both v4 and v6, maybe peering issue

  115. Menel

    It is already up again klaudie