XSF Discussion - 2017-09-21

  1. jonasw

    Zash, mhm thanks

  2. jonasw

    not sure what you expect me to do with it though

  3. Zash

    jonasw: me neither

  4. jonasw

    okay :)

  5. jonasw

    then I’m just going to ignore that for now

  6. Ge0rG

    Is there any way to mark a message as related to some other message, without specifying the exact type of relation? We have 0372 for explicit references, <thread>s, LMC, but they all imply a certain relationship

  7. jonasw

    what is the use-case?

  8. Ge0rG

    jonasw: MAM - when a given message is asked for, it is useful to provide all the related messages as well.

  9. jonasw

    I’m not convinced

  10. jonasw

    at least not as default behaviour

  11. Ge0rG

    jonasw: some messages only make sense on top of others, e.g. reactions or LMC

  12. jonasw


  13. Ge0rG

    jonasw: it would be impractical to deliver only the addon message but not the original one

  14. jonasw

    for reactions, I’d simply omit those

  15. jonasw

    (in the view that is)

  16. Ge0rG

    jonasw: if your client receives a reaction message with a dangling reference, you need to store the XML and process it later when the actual message is obtained.

  17. jonasw


  18. jonasw

    I don’t necessarily see an issue with that

  19. jonasw

    while handling out-of-order messages is more annoying

  20. edhelas

    the new CSS breaks horizontal scrolling on some XEPs https://xmpp.org/extensions/xep-0385.html

  21. jonasw

    scrolling works, but indeed it’s an issue that it’s required

  22. jonasw

    edhelas, can you make an issue on the xeps repository?

  23. Ge0rG

    385 is just too wide.

  24. edhelas

    jonasw done

  25. jonasw

    edhelas, thanks

  26. jonasw

    Ge0rG, we can’t go over all XEPs to fix that

  27. Ge0rG

    jonasw: we could define a common mechanism to cross-reference messages and use that?

  28. jonasw

    well, we could, but ...

  29. Ge0rG


  30. jonasw

    Ge0rG, sure, yet-another-standard? :)

  31. Ge0rG

    jonasw: stop spoiling the fun!

  32. edhelas

    it's possible in CSS to break lines properly

  33. jonasw

    edhelas, auto-breaking of lines in code examples sounds like a bad idea

  34. edhelas

    I also have a question

  35. jonasw

    Ge0rG, you’d want to rebase the existing use-cases (LMC etc.) on that, and especially with LMC I see a lot of breakage from that...

  36. edhelas

    I'm planning to use MAM to retrieve MUC histories, is it possible to disable the default messages pushed by the server on login ?

  37. daniel

    edhelas: yes

  38. edhelas

    oh ?

  39. jonasw

    edhelas, https://xmpp.org/extensions/xep-0045.html#enter-managehistory

  40. jonasw

    maxchars='0' is one example

  41. jonasw

    (I’d set maxchars='0' *and* maxstanzas='0' to be sure; I guess not all services support maxchars)

  42. edhelas

    oh, so I should explicitely disable them

  43. edhelas

    thanks jonasw :)

  44. jonasw

    np :)

  45. daniel

    jonasw: I think I only set maxchars and I never had problems

  46. SamWhited

    Ge0rG: https://xmpp.org/extensions/xep-0367.html

  47. SamWhited

    That was meant for things like that

  48. jonasw

    daniel, "never", not even biboumi? ;)

  49. daniel

    Even though maxchar is a very weird metric now that I think about it. I guess back in the day is just copy pasted that from somewhere

  50. daniel

    This will probably break if you set something other than 0

  51. jonasw


  52. daniel

    jonasw: didn't know that biboumi was a problem. Never got reported to me

  53. Ge0rG

    SamWhited: is there a reason for using message ID and not unique ID?

  54. jonasw

    odd, I had something about that in the back of my head. maybe it was about MAM instead

  55. daniel

    jonasw: yes. Mam did break the stanza queue for sm

  56. SamWhited

    Ge0rG: no, I don't think anyone supports unique ID

  57. Ge0rG

    XEP-0045 is full of quirks

  58. SamWhited

    But it doesn't matter to me

  59. Ge0rG

    I'd really love to fix message IDs instead of inventing new unique IDs. But backward compatibility!1!

  60. SamWhited

    Oh, am I mixing those up? Whichever. I would be fine coupling it to the unique stable message ID thing, whichever that is

  61. jonasw

    XEP-0359 (<https://xmpp.org/extensions/xep-0359.html>)

  62. SamWhited

    Right, that.

  63. SamWhited

    Yah, in that case the real reason this doesn't support it is that HipChat doesn't support it and I was never going to be able to convince them to let me ad even more stuff.

  64. SamWhited

    Ge0rG: Do you think you'd want to use it for something? If so I wouldn't mind updating it to use the unique ID stuff

  65. Ge0rG

    SamWhited: some weeks ago, the topic of grouping messages when delivering MAM data came up here

  66. moparisthebest

    just curious, how do you know messages are related?

  67. Ge0rG

    moparisthebest: that's the challenge

  68. moparisthebest

    like these might be in response to your stuff, in fact they are, but they might just be a random unrelated topic

  69. SamWhited

    Oh, this is for grouping messages sent in reply to another message? That seems different to me

  70. moparisthebest

    that seems like an unsolvable challenge from an automated perspective, so a xep to support the grouping seems useless

  71. SamWhited

    actually, maybe not…

  72. Ge0rG

    SamWhited: it's about finding out which messages are related to each other

  73. SamWhited

    I agree with half of what moparisthebest said; it seems unsolvable if you're trying to automate it. That doesn't mean you can't have a good UI that leads the user to do the work for you.

  74. moparisthebest

    I don't think you can and that's why it just returns the last X or whatever

  75. moparisthebest

    SamWhited, if I respond to you like this I guess you could call those grouped (by putting your name first)

  76. jonasw

    the relation is already spelled out in the XML (LMC, Reactions...)

  77. jonasw

    there’s just no unified way to detect it

  78. jonasw

    (it’s, from my understanding, not about generic replies)

  79. SamWhited

    moparisthebest: That seems like a sensible guess

  80. SamWhited

    This seems more like what <thread> is for though

  81. Ge0rG

    SamWhited: I'm not speaking about message that have content that relates to other messages

  82. Ge0rG

    SamWhited: I'm speaking about technical dependencies, like "this message is a reaction to message XY"

  83. Ge0rG

    so it doesn't make sense to parse / display the message without also having XY

  84. SamWhited

    oh yes, that's what I thought you meant originally, yah, that's totally what this is for

  85. SamWhited

    I think… you mean things like clients having a +1 button or a party hat button or something?

  86. Ge0rG

    so it would make sense, in the context of MAM, to use unique IDs

  87. jonasw

    SamWhited, yes

  88. Ge0rG

    SamWhited: yeah

  89. SamWhited

    Cool, yah, moparisthebest was confusing me :)

  90. SamWhited

    That is indeed what this was made for

  91. jonasw

    so let’s rebase xep-0367 on stable & unique message IDs and build a reactions XEP on top of that

  92. Ge0rG

    jonasw: +1

  93. Ge0rG


  94. jonasw

    you want me to do the work, right? ;P

  95. Ge0rG

    You still don't know the unique message ID of outgoing messages.

  96. SamWhited

    I don't remember the MID spec, but it allows clients to assign one too, right?

  97. jonasw

    SamWhited, indeed

  98. SamWhited

    ¯\_(ツ)_/¯ problem solved

  99. jonasw

    so called origin-id

  100. Ge0rG

    But MAM operates on the server-assigned ID

  101. SamWhited

    That seems fine; MAM operates on server assigned ID, this operates on client assigned ones.

  102. jonasw

    it would be convenient if the client could query MAM for the message if MAM doesn’t support grouping

  103. jonasw

    (the proposed grouping that is)

  104. jonasw

    alternatively, we could modify MAM to adopt the client-generated origin-id for sent messages iff it doesn’t conflict with any existing ID (and maybe reject delivery otherwise or something)

  105. Ge0rG

    I'm sure somebody will pull the "omg security!1!" card on that.

  106. moparisthebest

    Ge0rG‎: "this message is a reaction to message XY" -- how do you know that though?

  107. jonasw

    moparisthebest, reaction as in "+1 button"

  108. jonasw

    not as I react to your message right now

  109. jonasw

    Ge0rG, how?

  110. jonasw

    (and isn’t that usually your job?)

  111. moparisthebest

    he said 'this message is a reaction' not 'I hit a +1' (is +1 a thing in xmpp or any clients?)

  112. jonasw

    not yet, that’s the point

  113. jonasw

    and those "+1 buttons" are usually called reactions

  114. jonasw

    e.g. on github

  115. Ge0rG

    jonasw: let's see... if the server is using a typical hash-table implementation, a client could DoS it by contructing specially-chosen UUIDs.

  116. jonasw

    Ge0rG, meeeh :P

  117. jonasw

    just don’t tell clients your hash seed?

  118. Ge0rG

    This one! https://arstechnica.com/information-technology/2011/12/huge-portions-of-web-vulnerable-to-hashing-denial-of-service-attack/

  119. jonasw

    that’s a general vector, and afaict choosing a random hash seed is sufficient to fix that

  120. Ge0rG

    jonasw: I'm sure it's possible to extract the seed via side-channel attacks

  121. jonasw

    Ge0rG, *sigh*

  122. jonasw

    then we’ll have to operate on the client-generated ID. it’s saner anyways, because it works without MAM

  123. jonasw

    MAM will have to interpret these IDs then and intrenally store their relatedness for queries

  124. edhelas

    what is the correct request to get MAM history from MUC ?

  125. jonasw

    edhelas, ask the MUC for history?

  126. edhelas

    <iq xmlns="jabber:client" type="set" id="S8lR4X"><query xmlns="urn:xmpp:mam:1"><x xmlns="jabber:x:data" type="submit"><field var="FORM_TYPE"><value>urn:xmpp:mam:1</value></field><field var="with"><value>muc@conference.movim.eu</value></field></x><set xmlns="http://jabber.org/protocol/rsm"><max>300</max></set></query></iq>

  127. jonasw

    (I don’t know of any non-biboumi MUCs which support that)

  128. edhelas

    looks like I'm wrong somewhere

  129. jonasw

    edhelas, also, this may be better suited in jdev@

  130. Ge0rG

    jonasw: yeah. Because it's a good thing to have three different IDs on each message.

  131. edhelas

    a true

  132. edhelas


  133. jonasw

    Ge0rG, m(

  134. jonasw

    why exactly is a mam query a type='set' operation?!

  135. Ge0rG

    oh, wait. those IDs are all insufficient in some ways. Let's create a new XEP with a fourth, universal ID!

  136. jonasw

    Ge0rG, any other suggestion? other than getting a sent-carbon for your own message with the MAM ID >.>

  137. SamWhited

    hmm, I guess querying MAM for "a message and all of its reactions" would be sort of nice. I'm not sure if it's strictly necessary though.

  138. Ge0rG

    jonasw: that was exactly what I proposed back then.

  139. SamWhited

    It seems to add another dimension of complication for something that could easily be lived without.

  140. Ge0rG

    SamWhited: the problem is not "a message and all of its reactions" but rather "the message and all the messages you need to properly understand it"

  141. SamWhited

    Why would you ever actually need to do that? I don't dislike the idea of making sure that's queryable, it just doesn't seem like a common enough thing to worry about.

  142. Ge0rG

    SamWhited: if you receive a reaction message with an unresolved reference ID, you need to delay processing until you know what it is a reaction to

  143. SamWhited

    I suspect 90% of people will just sync history and you'll see new reactions appearing as history gets updated

  144. SamWhited

    That seems fine

  145. jonasw

    I tend to agree

  146. Ge0rG

    I tend to process messages when they are received, but that doesn't work with missing references.

  147. jonasw

    the argument Ge0rG makes is in the spirit of the "simple clients, complex servers" principle though

  148. SamWhited

    Does MAM not guarantee delivery order?

  149. SamWhited

    Oh, nevermind, stupid question

  150. jonasw

    ok :)

  151. SamWhited

    If you have a reaction and *then* you sync history it doesn't matter what MAM does.

  152. Ge0rG

    SamWhited: it only guarantees the order for the subset you request.

  153. SamWhited

    yup, yup

  154. SamWhited

    So in that case though you end up having to make a MAM query, then for each message make another query asking if there are any reactions you haven't received yet

  155. Ge0rG

    if there are any messages for a given reaction.

  156. jonasw

    SamWhited, no, the idea would be that the original MAM query would give you all messages to which there have been reactions in the result setd

  157. jonasw

    (even though they may be outside the range of messages you requested)

  158. Ge0rG

    jonasw: but that would violate message order, and lead to interesting results.

  159. SamWhited

    This seems like a lot of additional complexity when you could just store your reactions somewhere on the client and when you get a message if you have reactions in storage already display them.

  160. SamWhited

    Or when you get a reaction, if you have the message its linked to, also display it.

  161. Ge0rG

    example: you request 50 messages before #103. you get #53-#102 and #40 because it was reacted to. Next, you request 50 messages before #40, because it is the earliest in your history. #41-#52 are lost

  162. jonasw

    Ge0rG, all depends on the sanity of your implementation

  163. Ge0rG

    SamWhited: that makes (limited) sense if you store the original message XML somewhere

  164. jonasw

    Ge0rG, do you really need the XML? I don’t think so.

  165. SamWhited

    Ge0rG: I don't think the storage format matters

  166. Ge0rG

    SamWhited: but then you end up searching your storage for a reference to each new message you receive.

  167. SamWhited

    Ge0rG: That seems fine

  168. Ge0rG pulls the client DoS card.

  169. jonasw

    I pull the "use a proper storage backend" card

  170. SamWhited

    There is no client DoS card.

  171. jonasw

    e.g. a simple sqlite database

  172. jonasw

    just add a column "reaction_to" with a stable message ID thing

  173. jonasw

    add an index to it

  174. jonasw

    -> reasonable lookup time

  175. jonasw

    you also only need to do that for messages you fetch from the archive

  176. SamWhited

    Flat XML files are probably fine, you are going to get a bunch of stuff to store, that's just how XMPP works. If you're worried about scans and lookup times, its your job to do something intelligent and make sure you don't dos yourself.

  177. SamWhited

    Maybe that means creating an "unknown references" file to make scanning quicker, maybe it means using a database. That's up to you.

  178. Ge0rG

    I wish for the times when XMPP servers were complicated and XMPP clients were simple.

  179. SamWhited

    I agree with that in general, but this doesn't make the XMPP client complicated; it makes it a dumb display layer that doesn't have to have a ton of logic about doing crazy MAM queries.

  180. SamWhited

    It can just keep doing whatever it was already doing and process things as they come in. It might be processing the reaction, or it might be processing the message in terms of reactions it already has.

  181. moparisthebest

    do clients generally do mam queries reversed like that?

  182. moparisthebest

    conversations doesn't seem to, it seems to start loading from your last message and go forward

  183. moparisthebest

    in which case everything *just works*

  184. Ge0rG

    moparisthebest: I think there are two models of MAM usage: (a) query for everything since your last connection, (b) query a given channel / user history on demand when opening the tab

  185. moparisthebest

    oh I was waiting for b until I just realized gajim made that into a beer...

  186. Ge0rG

    "Clients and servers MUST NOT include an <attach-to/> element on messages with a non-messaging payload unless they are including it on an error which may be attached to the message that caused the error to be generated." What is a "non-messaging payload"?

  187. SamWhited

    No idea, no <body> I guess? That should probably be fixed.

  188. SamWhited

    Or just removed, I don't see the point in that restriction anymore.

  189. Ge0rG

    SamWhited: for reactions, I'd go with a separate non-<body> element, because people in here demanded invisibility to incompatible clients

  190. Ge0rG

    And because it will be awesome to have to change all the implementations that filter out body-less messages as "unimportant".

  191. SamWhited

    yah, just removing those lines from message attatching seems reasonable

  192. edhelas

    muc.xmpp.org doesn't supports mam:1/mam:2 yet ?

  193. MattJ


  194. Zash

    -version xmpp.org

  195. Bunneh

    Zash: xmpp.org is running Prosody version 0.9.12 on Linux

  196. moparisthebest

    -version muc.xmpp.org

  197. Bunneh

    moparisthebest: muc.xmpp.org doesn't reply to version requests

  198. edhelas

    yeah that seems to be the problem

  199. edhelas

    I don't see caps for muc.xmpp.org in my db

  200. moparisthebest

    Yea so we don't actually know what it's running :)

  201. Zash

    <identity type='text' name='Prosody Chatrooms' category='conference'/>

  202. Zash

    Impossible to know

  203. mathieui

    total mystery

  204. moparisthebest

    It could be lying

  205. moparisthebest

    Never trust a remote entity