jdev - 2023-12-17


  1. edhelas

    I'm curious to know how the current clients are requesting message history, is the MAM support server side good enough to use MAM + RSM fully ? Do you have a "load when scrolling up" requests ? If so, are you using before-id or start/end

  2. singpolyma

    I always use rsm

  3. singpolyma

    before-id was problematic IME and rsm is identical feature anyway

  4. edhelas

    > I always use rsm Works well on ejabberd and prosody ? So how does your request look like ?

  5. edhelas

    no start, end = date of your earliest message in the conversation and you add an empty <before/> to only get the last page ?

  6. edhelas

    With a limit of something like 50 messages per page ?

  7. singpolyma

    I mostly test on prosody. I use timestamp only fow initial load or when mam query fails

  8. singpolyma

    I mostly test on prosody. I use timestamp only for initial load or when mam query fails

  9. Zash

    singpolyma, implying that you leave out the timestamps when querying the next page?

  10. edhelas

    larma How does Dino handle this case ?

  11. singpolyma

    Zash: you mean if I started with a timestamp and then am getting next page in that query? I forget

  12. Zash

    I would expect requests for further pages to use the same MAM form parameters, but varying RSM.

  13. lovetox

    edhelas, you need to be aware that <flip-page/> and before-id is not supported by all servers

  14. lovetox

    i wonder if it is supported by ejabberd

  15. lovetox

    i have a feeling its not

  16. edhelas

    yes that's also why I'm asking how you guys figured out how to request MAM :)

  17. edhelas

    Especially the scroll-up-get-mam flow

  18. lovetox

    Gajim does not do backpage queries, but if you want to do it you can with rsm

  19. lovetox

    https://xmpp.org/extensions/xep-0059.html#backwards

  20. lovetox

    i dont think you can do any filters on the mam result set, simply supply nothing

  21. lovetox

    and only the rsm

  22. edhelas

    Mhhh but I have to take my earliest message as a reference before paging

  23. edhelas

    Because I don't want to resync what I already have

  24. lovetox

    yeah its like with after, you set the first id you have into the <before> element

  25. lovetox

    which gives you a page before that id

  26. lovetox

    be aware that you will not receive it in backwards order

  27. lovetox

    that would be what <flip-page/> would be for

  28. edhelas

    I'll check with RSM then :)

  29. lovetox

    if your server supports the new things, this will become a bit more natural, as for backwards paging you would supply <before-id> and <flip-page>

  30. lovetox

    and you would only decide on the page size with rsm

  31. lovetox

    hmm no thats not right, you still would use rsm <before>

  32. lovetox

    i think we did <before/after-id> to make it more natural to request holes in history

  33. edhelas

    One small question, to be compliant regarding the XMPP Compliance Suites, you don't need to fully implement the XEPs ? So for example servers that are not fully implementing MAM, they can still get compliants ?

  34. lovetox

    i would say so, this is a feature a server does not need to support

  35. lovetox

    would be weird to mark them as not compliant

  36. edhelas

    It's in Advanced Server indeed

  37. edhelas

    From what I see ejabberd doesn't implement the MAM 0.7.0 changes, I might open a ticket for that

  38. singpolyma

    > yeah its like with after, you set the first id you have into the <before> element Yes, this is what I do for scroll up. I don't use flip-page and I wouldn't have any reason to in my case either

  39. lovetox

    why? do you not display the message before the whole page is received?

  40. edhelas

    > why? do you not display the message before the whole page is received? For me I prefer to wait the end of the RSM page then update the UI in one batch

  41. singpolyma

    > why? do you not display the message before the whole page is received? Ripht

  42. singpolyma

    > why? do you not display the message before the whole page is received? Right

  43. lovetox

    yes then it does not matter

  44. lovetox

    and singpolyma do you store the message to a local db?

  45. edhelas

    Holger if I remember correctly we talked a few years back about before/after-id support in ejabberd, there was a technical limitation that would make things a bit complex to handle ?

  46. lovetox

    this would be suprising, this is simply a little bit different sql query

  47. edhelas

    ejabberd is having different DB backend

  48. lovetox

    whatever DB it is, would be a very weird db if you can turn around the operator for one field

  49. lovetox

    whatever DB it is, would be a very weird db if you cant turn around the operator for one field

  50. lovetox

    but maybe i learn something new now

  51. singpolyma

    > and singpolyma do you store the message to a local db? Yes

  52. lovetox

    and did you implement any measures for this ordering problem we talked about?

  53. singpolyma

    lovetox: i don't handle holes yet, so no

  54. lovetox

    it was not about holes

  55. lovetox

    but let me think for a minute

  56. lovetox

    yeah its about how you restore the order when loading messages from the db, if messages on the page start, and page end of the previous page have the same timestamp

  57. lovetox

    you would insert them in the wrong order into the db

  58. lovetox

    and somehow would need to reverse that on loading

  59. lovetox

    im talking about backpaging

  60. singpolyma

    Yeah I don't have any sub timestamp ordering stuff yet so I guess if they were both the same time it would be ordered unpredictably

  61. Zash

    And this is why bot replies are often shown before the command that triggers them šŸ˜‚ļø

  62. singpolyma

    I wrote a patch for prosody to get ordered mam IDs which would be nice, but of course nothing like that in the wild yet

  63. lovetox

    yeah i thought so, everybody basically says fuck it

  64. lovetox

    singpolyma, there would be a much easier way

  65. lovetox

    prosody simply supporting subsecond timestamps

  66. lovetox

    this reduces the chance of this happening to 1 in a billion

  67. Zash

    it does

  68. singpolyma

    Honestly I've been dogfooding my setup for a month and never ran into a wrong order, but I did plan to add a sequence number eventually

  69. lovetox

    since when Zash?

  70. Zash

    16 months ago according to the CHANGES file

  71. lovetox

    then i would say its impossible that even a bot can have message with same timestamp, because even prosody needs more than one microsecond to process a message

  72. Zash

    Not in a release of course

  73. singpolyma

    Zash: depends on storage driver right?

  74. Zash

    Sure

  75. lovetox

    > Not in a release of course ....

  76. lovetox

    like, next time you can add this to the sentence :)

  77. singpolyma

    This just comes out as fractional seconds in the delay timestamp?

  78. lovetox

    yes

  79. lovetox

    and then we dont need that sub ordering

  80. singpolyma

    I should double check my parser supports that but I think it does

  81. lovetox

    singpolyma, i looked into it when we talked about it, and deemed it to hard for me to implement

  82. lovetox

    i understand the theory, but the practice was crazy complex

  83. lovetox

    but all that mam ordering talk its simply solved if you have microsecond timestamps

  84. lovetox

    its practially impossible that a message has the same timestamp then

  85. singpolyma

    Close enough for most uses anyway

  86. singpolyma

    Even millis

  87. jonasā€™

    iff all your timestamps are from the same source.

  88. Zash

    not even then

  89. Zash

    time is wibbly wobbly, not to be trusted

  90. singpolyma

    It's true they aren't all, since live messages use local system clock

  91. lovetox

    > iff all your timestamps are from the same source. are you talking about some kind of distributed storage?

  92. jonasā€™

    ah true, somehow i was reading into the conversation that someone was putting sequential numbers into the microseconds :-)

  93. jonasā€™

    lovetox, no, I'm talking mixing timestamps from your local machine, your user's server, and potentially a MUC server in the same stream.

  94. jonasā€™

    I don't get why everyone has issues with non-timestamp ordering, but if I were to accept that, the best way forward would be if servers would just start to stuff a sequential ID into the microsecond field.

  95. jonasā€™

    (resetting the ID counter each second)

  96. lovetox

    i dont think people are against it, maybe some server developers deem it hard to implement

  97. jonasā€™

    that only gets funny when more than 1e6 messages are processed per second, and I don't think that's a realistic scale to happen for any single xml stream.

  98. lovetox

    from a client perspective it would be great

  99. Zash

    jonasā€™, rate limits to the rescue!

  100. jonasā€™

    lovetox, you're a *client* developer, aren't you?

  101. singpolyma

    > I don't get why everyone has issues with non-timestamp ordering, but if I were to accept that, the best way forward would be if servers would just start to stuff a sequential ID into the microsecond field. Welcime to uuidv7

  102. lovetox

    jonasā€™, yes?

  103. jonasā€™

    lovetox, and you insist on using timestamps?

  104. lovetox

    i didnt insist, it was more like, hey lets do a one line change in a server and request microsecond timestamp instead of second timestamp, brings us 99,99% to the goal

  105. Zash

    Is there a structure, maybe btreeish, where you can insert a thing before or after another thing and get a canonical order out of it?

  106. lovetox

    instead of waiting years until somebody slowly implements ordering

  107. jonasā€™

    what ordering?

  108. jonasā€™

    you get IDs, use them

  109. jonasā€™

    I don't see the issue

  110. lovetox

    im not sure im getting what you are trying to say

  111. lovetox

    ids are not orderable

  112. lovetox

    not sure how i can use them

  113. jonasā€™

    step 1: put IDs into your database' primary key

  114. jonasā€™

    wlel, probably not primary, because they're remotely assgined, but you get what I mean

  115. jonasā€™

    step 2: have a sequential number in your database

  116. jonasā€™

    step 3: use that _locally generated_ number for ordering

  117. jonasā€™

    step 4: use the IDs for querying the server

  118. Zash

    > i didnt insist, it was more like, hey lets do a one line change in a server and request microsecond timestamp instead of second timestamp, brings us 99,99% to the goal It was *far* from one line, FYI. Prosody is written in Lua which itself is written in portable C which does not have the concept of sub-second precision timestamps, nor formatting functions for it.

  119. jonasā€™

    step 5: done.

  120. Zash

    We have to deal with timestamp formatting functions that throw errors if you pass them floats.

  121. lovetox

    step 6, jonasā€™ tries to implement it and learns its not that easy :D

  122. jonasā€™

    I'd accept that challenge if there was an XMPP library in any language I care to deal with these days.

  123. lovetox

    because we are talking about backpaging, so you need a second locally generated id

  124. jonasā€™

    a second locally generated ID?

  125. jonasā€™

    in addition to the sequential one?

  126. lovetox

    a second sequential one

  127. Zash

    jonasā€™, it gets complicated if you start from now and work yourself backwards

  128. lovetox

    anyway, im not doubting that very good developers can do all that

  129. lovetox

    no xmpp client in existing doing that, maybe tells us something

  130. lovetox

    no xmpp client in existence doing that, maybe tells us something

  131. lovetox

    i take the timestamp ordering, works just as good

  132. lovetox

    and while i do that, wait for the server side ordered id

  133. MattJ

    edhelas [18:21]: > One small question, to be compliant regarding the XMPP Compliance Suites, you don't need to fully implement the XEPs ? So for example servers that are not fully implementing MAM, they can still get compliants ? Since I think this is about MAM, note that support for the new stuff is mandatory to be compliant with the latest XEP-0313, so yes. Compliance suites might need to indicate a version number.

  134. singpolyma

    jonasā€™: if you never do scrolling back your way is fine, but if you do then you might get new messages older than anything in your db. Or in very bad cases older than some things in your db and newer than other things. Even inside the same conversation. So your sequence number needs to be a logical tuple it's not a simple increment

  135. edhelas

    > edhelas [18:21]: > &gt; One small question, to be compliant regarding the XMPP Compliance Suites, you don't need to fully implement the XEPs ? So for example servers that are not fully implementing MAM, they can still get compliants ? > Since I think this is about MAM, note that support for the new stuff is mandatory to be compliant with the latest XEP-0313, so yes. Compliance suites might need to indicate a version number. Thanks !

  136. singpolyma

    Looks like an escaping bug snuck into your quoting again ;)

  137. edhelas

    šŸ˜¬