XSF Discussion - 2021-06-24

  1. Kev

    After the comment on the list about the carbons/MAM race condition, I was thinking further on this last night/this morning. It’s *really* unpleasant to solve. Easy to solve in protocol, horrible to try to actually implement said protocol.

  2. Ge0rG

    Kev: a carbons/MAM race condition post on the ML? I haven't sent any mails for months now!

  3. flow

    Kev, care to elaborate on the last part?

  4. flow

    is there some resource where the issues and potential solutions are listed? a wiki page? I know there were countless discussions of this, but sadly, the details a no longer in my memory (but probably in background storage somehwere ;))

  5. Ge0rG

    pick a random ML post by me about carbons and or mam ;)

  6. Kev

    Basic situation is that as a client you’d like to be able to login, enable carbons before receiving any messages and know the MAM ID directly preceeding the first message you will receive. (i.e. you don’t get dupes and you don’t get holes)

  7. Kev

    But on the server actually achieving that is rather unpleasant when the MAM query is going to be async, and you’re going to be continuing to receive messages for the user while that async operation is in flight.

  8. Kev

    s/MAM query/query to the backing store for MAM/

  9. flow

    ahh, so the atomicity of the protocol operation is hard to implement on the server side, without getting a performance hit. but I assume it's trivial on the client end

  10. Kev

    It’s trivial on the client end in the sense that the client doesn’t have to implement atomicity.

  11. flow

    I am not sure where the async part and "continuing to receive messages" comes from, though. can't you simply make it a synchronous operation?

  12. Kev

    I’ve come up with a relatively straightforward solution for a single node server, but haven’t got anything that works in a cluster yet.

  13. Kev

    Block the whole server until the db/fs result comes in?

  14. flow

    clarify "whole server"

  15. flow

    you block the inbound elements of the client performing the operation

  16. flow

    maybe even

  17. Kev

    That isn’t what’s needed.

  18. flow

    isn't this operation part of bind2?

  19. ralphm

    Is it a problem if there's an overlap?

  20. Kev

    ralphm: It could be construed a problem if the overlap causes dupes.

  21. ralphm

    But the dupes can be detected?

  22. Kev

    Unless we tell clients they need to dedupe, which would be …unpleasant.

  23. Kev

    flow: Yes, I was pondering how to actually implement bind2 in a non-trivial server implementation :)

  24. ralphm

    Well, we don't have exactly once delivery semantics, so shouldn't clients already do so?

  25. ralphm

    Generally clients also have local message storage and those need to be compared to MAM results as well?

  26. MattJ

    Kev: the client can't receive messages until the resource is bound, which is precisely why this should happen during binding

  27. MattJ

    Oh, but the user can, right

  28. Kev

    MattJ: Yes, that’s the point of Bind2, but that doesn’t make it easy to avoid the race on the server.

  29. Kev

    Yes, precisely.

  30. MattJ

    So after binding, reprocess any messages since the ID you gave the client?

  31. Kev

    What does ‘reprocess’ mean here?

  32. MattJ

    Pretend they just arrived, deliver them to the new client

  33. Kev

    By querying the archive, or by caching them in-memory in the server?

  34. MattJ

    That's up to you

  35. Kev

    Well, not really, because if you do it with an async operation (1), you get the same race again :)

  36. Kev

    But although I can solve this for the trivial case, I haven’t come up with a model that works for a multi-node server yet.

  37. Holger

    Right, it just moves the problem from the client to the server node.

  38. flow

    can't you even deliver them to the new client via the mam archive? not all messages are archived

  39. flow

    can you even deliver them to the new client via the mam archive? not all messages are archived

  40. flow

    so you have to do it by storing the messages in memory

  41. flow

    which probably means you need a defense mechanism in case the capacity exceeds a limit or a timeout happens

  42. flow

    or, erm, hmm, messages that are not archived are not relevant here…

  43. flow

    so you could do both, a in-memory cache and a mam query as fallback

  44. flow

    wait, the mam query races against new live messages, right?

  45. Kev

    You’re starting to get the picture of how this is less straightforward than it sounds, aren’t you? :)

  46. flow

    the sad thing is that I think I had that picture a while ago, but then lost it again

  47. ralphm

    Holger: FWIW, moving complexity to the server is a good thing

  48. Holger

    I do agree with that.

  49. Kev

    I agree with that to an extent. Implementation complexity I agree with. Pushing computational complexity onto the server adds up pretty fast.

  50. MattJ

    Unrelated: does anyone recall which spec contains the MIX roster annotation stuff?

  51. MattJ

    Aha, XEP-0405

  52. MattJ

    That's confusing

  53. MattJ

    Ok, so XEP-0405 is a candidate to replace or merge into PAM (XEP-0376)

  54. arc

    Did it connect?

  55. arc

    Oh finally. !

  56. arc

    Ralphm: you here?

  57. Sam

    o/ if you meet today please consider approving or sending feedback to the initial fiscal host rules (and bear in mind that we can always change them later, we just need something on the website to make opencollective happy): https://github.com/xsf/xmpp.org/pull/915

  58. arc

    I just spent 4 hours setting up an alternative xmpp host along with updating all my SSL and let's encrypt pieces.. so I certainly hope we are meeting today!

  59. arc

    It appears that something in the let's encrypt API changed and that threw out all my Cron tasks for monthly cert updates.

  60. arc

    MattJ, dwd: ?

  61. MattJ

    Here o/

  62. MattJ

    Not looking hopeful :/

  63. arc

    No it's not.

  64. dwd


  65. dwd

    Sorry, had some networking issues.

  66. dwd

    MattJ, arc ^^

  67. arc

    Join the club 😆

  68. MattJ


  69. arc

    Well we are missing a chair

  70. arc

    ralphm: is apparently away this week

  71. Zash

    Be the chair you wish to see

  72. arc

    I have two sick hens and just spent 4 hours trying to get lets encrypt to work again, so not it

  73. dwd

    Well, this is fun. I can see inbound messages on Gajim on my laptop but not reply, but if this works I can send but not receive on my phone.

  74. arc

    You are received!

  75. dwd

    Man that's weird.

  76. dwd

    So, anyway. I've yet to review the fiscal policy, and in particular I don't know if it should be a XEP to give us the change process.

  77. MattJ

    Only if I can append a cookie recipe

  78. dwd

    I do need to do a pass over the CoC in order to pick up comments.

  79. dwd

    And yes, a cookie recipe, I'm totally up for that.

  80. MattJ

    There is also another document for review: https://github.com/xsf/xmpp.org/pull/933

  81. dwd

    Ah, I had not seen that.

  82. dwd

    On a quick skim, I'm mildly concerned by the notion of declaring an Experimental Procedural XEP to be, in effect, Active.

  83. MattJ


  84. dwd

    I understand why, but it has risks.

  85. MattJ

    The rationale is that the scope is just operators@, for the window of time until the CoC advances

  86. MattJ

    and it avoids a copy/paste into that document

  87. MattJ

    We've had more need of moderation this week, and I'd really like to get this document up

  88. MattJ

    I don't even know that it *needs* board approval, I'd happily host it on my personal domain and link people to it there while I'm the one moderating the MUC

  89. dwd

    Understood. I won't block it, and as I say I understand the desire to get something up ASAP.

  90. dwd

    And indeed, I don't know that it needs Board approcal, but it certainly doesn't hurt. It's a well written document, I'd be happy for it to be on our site, and governing the operators@ channel.

  91. MattJ

    Yes, I'd obviously rather board approval and having it on xmpp.org - otoh if we don't approve it this week and further need arises, I would rather just merge it or host it elsewhere meanwhile

  92. dwd

    arc, You have an objection to just merging it now?

  93. dwd

    arc, Sorry, that sounded leading. I mean, do you have any objection?

  94. arc

    I do not have any objections no

  95. arc

    I think at this point the more eyes that are on it the better

  96. MattJ


  97. MattJ

    Obviously it's not set in stone... if people are in favour of the document in principle we can (and likely will) amend it down the road

  98. dwd

    Right, we seem to have run out of steam. I'll get on with a CoC update tomorrow and hopefully move that along toward a Last Call.

  99. arc

    Sounds good. +1 week?

  100. MattJ


  101. dwd

    Sounds good to me.

  102. moparisthebest

    since the meeting is over, relevant to earlier dwd + arc https://www.moparisthebest.com/images/xmpp-vs-matrix.jpg (I couldn't find original so I remade it from memory)

  103. ben

    heh that's very accurate

  104. edhelas

    moparisthebest :D

  105. edhelas

    moparisthebest time to put it on /r/xmpp

  106. mdosch


  107. jubalh

    moparisthebest: :D

  108. jubalh

    i will have to steal this for twitter

  109. eevvoor

    emus, I made a pull request for the xmpp provider list.

  110. eevvoor

    Due to the blabber.im takedown in three days.

  111. emus

    thx, I recoomend to let the emotions calm down a bit. Seem they change their minds quickly

  112. eevvoor


  113. eevvoor

    Not good for server admins to be emotional.

  114. emus

    well, everyone is emotional, the responsibility is to get to know all of them that you know how to deal with it accordingly