jdev - 2023-07-19

  1. thomaslewis


  2. lovetox

    i still contemplating how MAM requests of random ranges can be done on a client that stores the messages locally, while maintaining message order with messages who have the same timestamp

  3. lovetox

    it seems not possible currently, prosody for example does not have fractual timestamp support

  4. lovetox

    i would say in a Groupchat its very likely that messages end up with the same timestamp

  5. Zash

    You can't trust timestamps, regardless of precision.

  6. lovetox

    if we take bots into consideration it gets worse

  7. Kev

    Not entirely sure what you're asking, but I think as long as you store 'holes', you can always keep your local archive in order.

  8. Zash

    Clocks can go backwards do all sorts of funky things thanks to NTP

  9. Zash

    Clocks can go backwards and do all sorts of funky things thanks to NTP

  10. lovetox

    Zash, thats not really my problem, if a server archive communicates a timestamp, i will show it to the user in that order, of course a timestamp can be wrong on the server, but its not my problem as a client

  11. lovetox

    servers are certainly able to keep the correct time, its not mission impossible

  12. lovetox

    Kev, if i store something in the database, i need some field i can use for the order

  13. lovetox

    normally this is timestamp, and a second category, insertion order/entity key

  14. lovetox

    for the case where you have 2 messages with the same timestamp

  15. singpolyma

    You're saying you want MAM sequence numbers?

  16. singpolyma

    So everyone gets the same order from MAM?

  17. Zash

    Pray that you never encounter someone hosting on a raspberry pi or something without a proper clock, that might always have the clock set to 1970 every time it boots.

  18. lovetox

    no, i describe a problem, and want to ask if someone has a solution

  19. singpolyma

    Ok, what is the problem?

  20. Zash

    The solution is not to use timestamps, that just gets you different problems.

  21. lovetox

    think about requesting MAM backwards, in small increments

  22. lovetox

    say 10 messages now, then another 10 messages and so on

  23. lovetox

    you will end up with a message in each query, but with the same timestamp

  24. lovetox

    basically you request between those 2 messages that have the same timestamp

  25. lovetox

    but because you request backwards, now the older message is later inserted into your database

  26. lovetox

    you cant order on timestamp, and you cant order on entitykey

  27. singpolyma

    Right, so it's what I said and you want consistent ordering for mam users?

  28. Zash

    Isn't this one of the Hard problems in distributed computing? :)

  29. singpolyma

    Zash: if it's the problem I think it is (ordering) then for muc it's not hard because we have a choke point

  30. lovetox

    ok so are you saying there is no solution that comes to mind, except changing the standard to make it possible?

  31. lovetox

    its a known limitation that i hit or what?

  32. singpolyma

    If the problem is the one I said then yes, MUC (and also MUC mam) do not guarantee ordering right now

  33. Zash

    You do get order implicitly from MAM. If you were to insert it into a linked list it should be easy to preserve that order.

  34. Zash

    So this problem is actually SQL.

  35. singpolyma

    Zash: if the mam from the server was always exactly the same order you could synthesize sequence numbers client side, that's true

  36. Zash

    Deriving sequence numbers gets messy but not impossible. Hence if you were to insert into a linked list (just a sequence, no sequence ids), things remain easy.

  37. lovetox

    Zash its trivial to communicate a sequence number for a single server, you are talking about some edge case where someone distributes its database across X servers or not?

  38. lovetox

    and in that case simply dont communicate ones

  39. Zash

    I'm just talking about replicating order on the client doing MAM requests

  40. lovetox

    if you are a single server, and have fractual timestamp support

  41. lovetox

    then you can make sure its impossible that 2 messages are inserted with the same timestamp

  42. lovetox

    and then you have naturally a sequence

  43. lovetox

    or you simply communicate the sql row id, or whatever your entity key is

  44. lovetox

    i guess you insert messages in the order you receive it

  45. lovetox

    as you have to recreate that order when someone queries the messages

  46. Zash

    there are no guarantees that a computer clock will always produce an increasing number

  47. Zash

    unless you use the monotonic clock, but that usually does not have anything to do with wall clock

  48. Zash

    my education is in watchmaking and I am saying: do not trust timestamps for anything but presentation

  49. lovetox

    yeah, and thats what we are talking here, presentation

  50. Zash

    I thought you were talking about ordering

  51. lovetox

    so why not use a sortable entity key?

  52. lovetox

    then you have your guaranteed increasing number

  53. lovetox

    or simply a field that autoincrements

  54. Zash

    Prosody with default storage does in the form of a sequence number, or possibly a file offset, in the list files it stores archives in.

  55. Zash

    But that resets every time messages are removed.

  56. Zash

    The SQL driver also has a hidden sequence number, but telling you that would give you hints about how many messages other people have received, and that seems a Bad Thing we should avoid.

  57. Zash

    Prosody is getting high precision timestamps, but I don't condone their use for ordering.

  58. lovetox

    what if i somehow include in my querys as filter the timestamp

  59. lovetox

    and if that query completes i know i have received everything from this timestamp

  60. msavoritias

    what should a client do instead of ordering though?

  61. msavoritias

    reading the xep it just says dont use timestamps and the server should try to preserve order thats it

  62. msavoritias

    but then it says

  63. msavoritias

    > The collection is ordered chronologically by the time each message was sent/received.

  64. msavoritias

    so back to timestamps

  65. msavoritias

    so i guess i am missing some unique id within a group chat that identies a message and its position in the graph of the group chat?

  66. msavoritias

    or am too optimistic

  67. Zash

    There's no graph, this isn't Matrix

  68. Zash

    MAM is an interface to a big ol' array

  69. msavoritias

    i said graph because briar does it as a graph

  70. msavoritias

    > The message graph is a directed graph where each message points to its dependencies. The list of dependencies is encoded in the message body in a format determined by the client. Since messages are immutable, dependency relationships are also immutable and the graph is acyclic. The graph may have more than one component.

  71. Zash

    You could have a pointer to the previous message and use that to detect holes and derive ordering, but nobody has written the XEP for that yet.

  72. msavoritias

    but since you said matrix i am guessing briar is closer to the matrix architecture than the xmpp one

  73. msavoritias

    > You could have a pointer to the previous message and use that to detect holes and derive ordering, but nobody has written the XEP for that yet. ah. so we are missing a correct way to detect ordering of messages with mam. I will add it to my list 🙂

  74. Zash

    But as singpolyma reminds us, messages go trough a single point where they are archived in an order. That order is what you get back out through MAM, but it's implicit.

  75. msavoritias

    yeah i read that. it seems to put a lot of trust that the server is actually working correctly though /thinking

  76. Zash

    msavoritias, if you are querying MAM you get order and everything

  77. Zash

    This is XMPP! Servers are always correct! // Server developer.

  78. msavoritias


  79. msavoritias

    > msavoritias, if you are querying MAM you get order and everything implicity as you said. because we trust in MAM \o/

  80. lovetox

    if you assume a broken server, then you have lost anyway

  81. lovetox

    what are these magical broken servers, that store all messages correctly but fail to afterwards send them correctly, and only the genius design of mentioning the previous message, saves the day

  82. Zash

    XMPP servers are never broken. But clocks and timestamps come from the domain of the Kernel and the hardware, and are thus unreliable and subject to physics being weird.

  83. singpolyma

    If your want consistent order and assume the server delivers consistent order then you can make sequence numbers client side if needed. This doesn't help with live messages

  84. lovetox

    singpolyma, how would that work with the example i presented

  85. Zash

    Live messages always have the timestamp 'now' unless specified otherwise with some delay tag.

  86. lovetox

    no server validates delay tags, they can safely be ignored

  87. singpolyma

    Zash: right, but in terms of consistent ordering vs mam there's no way to do it with live messages. But probably it's usually fine

  88. moparisthebest

    What's to validate?

  89. lovetox

    that it comes from the source it says in the by attribute

  90. singpolyma

    > singpolyma, how would that work with the example i presented Querying backwards? Negative sequence numbers at least. Depending on need possibly a tuple

  91. lovetox

    mhm, i will think about that, thanks

  92. moparisthebest

    I'm not sure what you mean, you are getting a message from some remote source along with when they said they sent it, nothing to validate there for anyone no?

  93. lovetox

    of course, its something different, when the muc server sets the delay, then a remote user

  94. lovetox

    depending on which set it, i trust it in different ways

  95. lovetox

    or lets say, i would, because currently its generally untrustworthy

  96. moparisthebest

    When does "trust" ever come into it, it's purely display I think

  97. moparisthebest

    "The sender of this message said they sent it X" that's all

  98. lovetox

    yes and in which order messages arrive, can be very serious

  99. moparisthebest

    When they arrive is a totally different matter

  100. lovetox

    thats the way if you dont trust it

  101. lovetox

    then you simply display "X said Y" take it or leave it

  102. lovetox

    if you trust it, i sort messages differently

  103. lovetox

    like i trust a MAM timestamp

  104. lovetox

    i dont tell the user, hey MAM said this was the time the message arrived, but i dont trustit

  105. lovetox

    its mainly about if you order after user-delay in the view or not

  106. lovetox

    im generally not a fan of reordering on user-dealy

  107. moparisthebest

    Personally I like messages sorted by arrival time but timestamped the date the sender said they sent them but eh

  108. lovetox

    im generally not a fan of reordering on user-delay

  109. lovetox

    but i could imagine it if the delay was set by a server

  110. Zash

    We don't have a thing for "date the sender said they sent them"

  111. lovetox

    its simply about validating the by attribute

  112. lovetox

    like you do with mam messages

  113. lovetox

    ahh stanza id i mean

  114. Zash

    We have "a thing on the network path failed at Instant Messaging"

  115. moparisthebest

    I think Conversations inserts timestamps always? I swear I've seen this behavior but I can't verify right now

  116. lovetox

    i heard that too

  117. singpolyma

    Not always

  118. singpolyma

    But on delayed send yes

  119. Zash

    _Delayed_ delivery, yes.

  120. lovetox

    point is, the whole request MAM in MUC is a non-trivial task

  121. lovetox

    it cannot be expected from your average developer that writes a new xmpp client

  122. lovetox

    would be great if the community could come up with some solution

  123. lovetox

    the making a sequence manually in the client is though a idea, although i have to store a range per archive