XSF Discussion - 2018-08-20

  1. Zash

    Does anyone remeber what the point of https://xmpp.org/extensions/xep-0292.html#self-iq was?

  2. Maranda eyes anjan broadcasting one presence any second 🙄

  3. Ge0rG

    What's wrong with that? MUCs are made to scale!

  4. Maranda

    Ge0rG: but my eyes don't scale 😞

  5. Ge0rG

    Maranda: are you reading the XML log?

  6. Ge0rG

    ...and only see the blondes, the brunettes, ...

  7. Maranda

    Ge0rG: trying to catch something from the log tail actually 🙄☺️

  8. Maranda thinks: luckily now CSI deduplicates 😆

  9. Ge0rG

    depending on the CSI implementation and violation of RFCs.

  10. Maranda thinks he doesn't break in order processing.

  11. Maranda

    Although staying in CSI theme...

  12. Maranda

    "For example: A client sends a CSI active nonza, followed by an XMPP Ping request to the server. The server first changes the CSI state to active and flushes all eventually queued *stanzsa*."

  13. Ge0rG

    Maranda: but you are dropping stanzas

  14. Maranda

    Ge0rG, so whenever I filter a stanza silently I'm breaking in order processing...?

  15. Maranda

    Perhaps everyone violates rfcs

  16. flow

    Ge0rG, Does the RFC forbid that?

  17. Guus

    The RFC states that stanza order MUST be preserved.

  18. flow

    Yep, but I meant "dropping stanzas"

  19. Maranda

    Ge0rG, so since *everyone* does that I guess there isn't a real problem isn't it? :P

  20. Guus

    dropping != reordering, I believe.

  21. Maranda

    or you can't filter and drop stanzas anymore

  22. Maranda

    Guus, I'm not reordering

  23. Guus

    I'm not saying you are. 🙂

  24. Maranda


  25. Guus

    I do wonder how effective CSI can be though, given that there's bound to be a stanza that you don't want to queue, and you need to maintain stanza order.

  26. Guus

    doesn't give much room to play, I think

  27. Guus

    at least not for "drop all but the last" type of filtering.

  28. Maranda

    Guus, I'm just queuing presences so it's... rather easy to do

  29. Guus

    Maranda, sure, but as soon as you get an IQ ping, you'd have to flush all that, right? Before responding to the IQ ping, I mean.

  30. Guus

    (or any other non-queued stanza)

  31. flow

    Guus depends on who send the ping. but yes, usually you would flush if you have to send something anyway

  32. Maranda

    Guus, hmm that I didn't think about.

  33. Guus

    flow, an inbound ping to the client, I mean. I don't think it matters much who sent the ping? The server needs to preserve the order of all stanzas, not depending on who is the originator?

  34. Ge0rG

    So you want to tell me it is okay with the RFC to drop stanzas, but not ok to reorder them?

  35. flow

    Guus, I believe that XMPP could fall into the category where the order only matters with regard to two endpoints, not necesarily globally

  36. Guus

    flow: I'm not sure if that works. How would that work with private messages in MUCs, for example

  37. flow

    So if the ping originated from an entity which has no queued stanzas, you could only deliver the ping without flushing (but there is no reason you shouldn't flush)

  38. Guus

    Ge0rG: I'm not sure, to be honest. My point is even if it were allowed, I wonder if there's much practical benefit to CSI.

  39. Guus

    got to pick up kid from school. bbl

  40. MattJ

    Guus: flow: in practical terms the order between two bare JIDs must be preserved

  41. Guus

    Ge0rG: but isn't that how privacy lists work too, btw?

  42. flow

    MattJ, why bare JID and not just "JID"?

  43. MattJ

    flow: imagine reordering presence from a MUC join, which specifies you always receive your own presence last

  44. Maranda is rather confused now.

  45. MattJ

    In my opinion servers need to be very careful if they start arbitrarily dropping or reordering stanzas

  46. Ge0rG

    What MattJ said.

  47. Maranda

    well if it's bare jids IQs are excluded from the equation I suppose.

  48. Maranda

    clients won't have to answer to those.

  49. Kev

    Reordering is worse than not delivery

  50. MattJ

    Maranda: I didn't mean it like that

  51. Kev

    Reordering is worse than not delivering

  52. MattJ

    But I'm on my phone so I don't think I can attempt a better explain

  53. Kev

    (Note that 'not delivering' isn't the same as 'dropping' - if you don't deliver, you still have to follow the rules for when to silently drop and when to error)

  54. Ge0rG

    My point is: if the RFC mandates in-order delivery, it is kind of implicit that it mandates delivery.

  55. Ge0rG

    So just dropping stanzas will violate the RFC

  56. Maranda

    ... and so everyone in a way or another violates it.

  57. Kev

    Ge0rG: Delivery or error, yes.

  58. MattJ

    It's my understanding that it mandates delivery

  59. MattJ

    Or error

  60. Ge0rG

    Some cases allow silent dropping.

  61. Zash

    But XEPs can override core rules?

  62. Ge0rG

    Zash: yes

  63. MattJ

    Depending on stanza type, as you well know

  64. Kev

    Zash: Yes, by negotiation.

  65. flow

    MattJ, yep, the xep45 self-presence order is a good point

  66. Maranda

    Perhaps you're suggesting to do some magic amendment to the CSI spec?

  67. Maranda


  68. Maranda isn't sure about how much he doesn't violate anymore, although he doesn't reorder what's in the queue.

  69. MattJ

    Maranda: the CSI spec does not tell you to reorder or drop

  70. flow

    Maranda, some ppl will tell you that you violate the RFC, some will tell you possible the opposite. What matters is: Is it beneficial? Does it break things?

  71. MattJ

    Unfortunately it has some "examples" which I am not happy about, but consensus was to add them

  72. MattJ

    They are not part of CSI

  73. MattJ

    And they are not specified in any detail, which may be worse

  74. flow

    Then get rid of them from the XEP. But we should consider adding a wiki page for CSI where strategies are collected

  75. flow

    Working and non-working strategies that is

  76. Zash

    Pros and cons

  77. MattJ

    flow: iirc you were one of the main voices requesting their addition :)

  78. Zash

    And whys

  79. Maranda

    flow, it's very beneficial and doesn't break anything tbh.

  80. MattJ

    Maranda: how do you know it is beneficial?

  81. Maranda

    MattJ, deduplicating presences?

  82. MattJ

    This is another problem I have with current CSI usage

  83. Maranda

    MattJ, well it reduces queues *a lot* expecially if you join a lot of big mucs.

  84. MattJ

    Maranda: OK, spammy broken clients

  85. MattJ

    But the MUC could filter that

  86. MattJ

    More simply

  87. Maranda

    MattJ, irregardless of broken clients

  88. MattJ

    bbl, train arriving at station

  89. flow

    MattJ, wait, we are talking about the in order requirement § 5.1? I thought we talk about CSI strategies that are applied

  90. Maranda

    flow, I guess that if you queued all stanzas there wouldn't be problem with in order processing, the issues come out only if you don't filter all routable payload but only one or two of 'em.

  91. Maranda

    flow, I guess that if you queued all stanzas there wouldn't be problem with in order processing, the issues come out only if you don't filter all routable payloads but only one or two of 'em.

  92. Maranda

    (like I currently do)

  93. flow

    Maranda, figures would be interesting. like the bytes that are not send by your implementation

  94. Holger

    Guus: > I do wonder how effective CSI can be though, given that there's bound to be a stanza that you don't want to queue, and you need to maintain stanza order. For me in practice there's often long periods where many presence stanzas are received, and nothing else.

  95. Maranda

    flow, well for my mobile account the average queue for staying inactive was like 600 presences without dedup, it went down to as little as 50-60.

  96. Maranda

    flow, well for my mobile account the average queue for staying inactive few hours was like 600 presences without dedup, it went down to as little as 50-60.

  97. flow

    Maranda, so that is CSI with and without queue (presence) dedup?

  98. Maranda

    So I guess that matters somehow, about the actual bw you save I'm not sure but you can make an esteem.

  99. Maranda


  100. flow

    k, I was more wondering about CSI vs non-CSI

  101. Maranda

    so there's for sure a benefit expecially on MUC usage.

  102. flow

    and especially the longest time period the connection was idle

  103. flow

    time for mod_csi_stats

  104. Maranda

    well I suppose you can extract those numbers for my figure if the server queued around 600 presences mostly for mucs during inactive hours

  105. Holger

    flow: Dedup can of course make a great difference if you limit the queue size and flush when it's reached.

  106. Maranda

    well I suppose you can extract those numbers from my figure if the server queued around 600 presences mostly for mucs during inactive hours

  107. Maranda

    for sure the reduction is reasonably around x10 magnitude

  108. Maranda

    sending only the last useful presence state

  109. flow

    Holger, good point

  110. dos

    heck, it doesn't need MUCs - I have an account with 700 transport contacts from a network with lousily defined presence semantics and CSI helps there as well in taming the unnecessary traffic

  111. flow

    dos, which network is this?

  112. dos

    facebook :P

  113. Ge0rG

    even grouping the presence XML stanzas over time and sending them in one big flush is good for the battery.

  114. Maranda

    Ge0rG, yes which is where deduplication helps...

  115. Holger

    > <MattJ> flow: imagine reordering presence from a MUC join, which specifies you always receive your own presence last I guess the problem with breaking this is that the client might mis-interpret presence received after the final self-presence as *newly* joined members?

  116. MattJ


  117. MattJ

    dos: any actual stats?

  118. Zash

    Join a few noicy IRC channels

  119. Ge0rG

    Holger: imagine how that interacts with GC1.0 joins and intermittent s2s outages.

  120. Zash

    There's no GC1.0 la-la-la can't hear you

  121. Ge0rG

    I've done a CSI benchmark on a freenode transport some two years ago.

  122. Ge0rG

    I prepared a motion to Council to get rid of GC1.0, but it was not approved.

  123. Ge0rG

    And when I started making a PR to 0045, I realized that significant deletion is required

  124. Zash

    Next Prosody won't allow GC1.0 joins, have we seen any ill effects of it yet?

  125. Holger

    So you basically cannot de-dup MUC presence without breaking that 0045 guarantee unless you add special code for handling MUC joins.

  126. Zash

    Holger: Mmmmm, special cases

  127. Holger

    They're so elegant!

  128. dos

    MattJ: I haven't made any stats so far, but when I hacked up my Nokia N900 to actually use CSI, it was the first time I could actually observe prolonged silence on the stream

  129. Zash

    Specal casing on top of hacks on top of more hacks until your brain explodes

  130. dos

    I've got around 60 presences in the last few minutes

  131. Maranda

    fwiw I didn't yet see a single muc break.

  132. Holger

    Maranda: Do you have a CSI client that shows MUC joins/leaves?

  133. Holger

    Without one I'm not sure how you'd "see" that breakage.

  134. jonasw

    and even then it might actually be hard to notice

  135. Maranda

    Holger, join a small channel compare occupant list with another client?

  136. dos

    you're likely to join MUCs when CSI indicates "active", for instance

  137. Holger

    Maranda: The resulting occupant list will be correct in both cases.

  138. Ge0rG

    dos: another good one is client caps caching

  139. Maranda

    I'm sure that works, and I just did that because I was worried something like that happened because of deduplication.

  140. Maranda

    Holger, I didn't mean breaking the spec :P

  141. Holger

    Maranda: AFAICS the only problem would be bogus "Maranda joined" messages.

  142. Maranda

    I meant *it actually* broke anything.

  143. Ge0rG

    dos: my client will join MUCs when inactive, because a MUC ping timeout happened when idle

  144. dos

    Ge0rG: yup. thankfully, that was already done by the whichevet old Telepathy version is there :)

  145. Holger

    Maranda: Yes I think it doesn't *actually* break anything because Conversations & friends don't show "Maranda joined" messages anyway.

  146. Maranda

    Holger, I'd suppose clients look more at status 110 now than the order.

  147. Maranda

    for self presence

  148. dos

    Ge0rG: and how likely are you to notice some erroreneous joins displayed by your client in such case? ;D

  149. Holger

    Maranda: That statement makes no sense to me.

  150. Ge0rG

    dos: there are issues.

  151. Holger

    Maranda: The point is, when they receive their status-110-message they assume they have the full list of occupants. Any presence received afterwards would be interpreted as "Maranda joined", while in reality Maranda didn't join but was an occupant before I joined.

  152. Kev

    Am I being completely daft, or does https://tools.ietf.org/html/rfc6121#section-8.5 completely fail to say how to route presence type=error?

  153. Maranda

    Holger, hm ok now that's clearer :P

  154. Maranda

    I was more looking at actual occupant list desyncs rather than that anyways

  155. Holger

    Yes those shouldn't be broken by this kind of dedup.

  156. jonasw

    only temporarily

  157. Holger

    Well yes only until the client processed the flushed stanzas.

  158. jonasw

    or if your dedup has a bug

  159. Holger


  160. Holger

    I haven't heard of bugs in XMPP code.

  161. jonasw

    that’s because they eat the messages telling you about them

  162. Maranda

    Holger, but question is how many mobile (one presumes most using CSI are those) actually show MUC joins?

  163. Maranda

    Holger, but question is how many mobile clients (one presumes most using CSI are those) actually show MUC joins?

  164. dos

    Maemo 5 does, and I modified it to use CSI, so I guess I could test it with ejabberd :v

  165. Maranda

    dos: doo eet 😁

  166. Holger

    Maranda: Yes, as I said :-)

  167. Holger

    Maranda: The answer may well be zero.

  168. Holger

    Maranda: I'm doing that kind of dedup as well as the possible benefit seems significant to me, and I haven't seen any (reports on) breagage. But this *is* hairy of course.

  169. Holger

    "breagage", I'm drunk.

  170. Maranda

    Holger, ditto I mean as far as I like strict compliance, the current CSI code works and is (surprisingly) clean and simple for once, so as long as something doesn't break (e.g. like occupant list desync) I'm not inclined to touch it 😁

  171. jonasw

    MattJ, could you kick anjan temporarily? apparently, they’re flooding this room with useless <presence/> stanzas

  172. MattJ


  173. MattJ

    An unsurprising outcome, in hindsight

  174. Ge0rG

    GC1.0 FTW!

  175. Ge0rG

    Unfortunately, there is no way to burn GC1.0 without "breaking" usability.

  176. Maranda

    damn I actually needed anjan to test something

  177. Ge0rG

    Maranda: get yourself kickbanned as well, then you are together with anjan again 🤔

  178. Maranda

    Ge0rG, :P

  179. Maranda

    Ge0rG, no need it actually works.

  180. Maranda

    And that problem wasnt gc1.0 related me thinks just an old issue in gajim that made it loop resending muc presences

  181. Ge0rG

    Maranda: but with GC1.0, the ghost returned after getting kicked

  182. Maranda


  183. Andrew Nenakhov

    Btw, are there any docs on GC1.0? We didn't find it anywhere, only references in XEP-0045

  184. dos

    hmm... when client receives a message directed to other client via carbon that has a delivery receipt request, should it send one?

  185. daniel

    That can make sense

  186. dos

    nvm, xep mentions it, so I guess it should: "Because it is possible for a given content message to be delivered to multiple XMPP resources controlled by the recipient, the sender of the content message needs to be prepared to receive multiple ack messages."

  187. daniel

    The xep is probably not talking about carbons though

  188. daniel

    But still I belive sending them - while not ideal - is the best thing we can do for now

  189. dos

    I think this is the only mention of a situation with multiple acks, so I'd interpret it as "it is possible, for instance with bare JID messages, message carbons are enabled etc."

  190. dos

    so nothing really mandates you to send them, but conforming receipt requestor should be prepared to handle them, so it should be fine

  191. Andrew Nenakhov

    I'd rather send it. Even if sender didn't intend my device to receive this message, I still received it, because my carbons are configured this way. So, it's delivered to me, and I should probably send a receipt.

  192. Andrew Nenakhov

    I also don't like when remote party dictates how my devices work. "I'll talk with you over this Android phone, not over that desktop client"

  193. jonasw


  194. Ge0rG

    you could send a carbon of a receipt 😁

  195. dos

    Psi does it even better right now - it sends a receipt for an outgoing carbon that requests one :D

  196. Ge0rG


  197. Ge0rG

    dos: as long as it doesn't request a receipt for a receipt, everything is good.

  198. Zash

    Something something two generals problem

  199. lovetox

    in my opinion of course a receipt should be send

  200. lovetox

    in my opinion of course a receipt should be sent

  201. lovetox

    a carbon means only the server received the message, not the client

  202. Ge0rG

    lovetox: what?

  203. lovetox

    what part of what i said is not clear?

  204. Ge0rG

    > a carbon means only the server received the message, not the client

  205. Ge0rG

    maybe I'm just lacking the proper context for this

  206. lovetox

    you receive a "received" carbon, now you can wait 5 seconds or any amount of time you want, if there is no receipt coming in from your other client, you should assume he didnt receive the message

  207. lovetox

    and ack

  208. Ge0rG

    I'd just ack anyway

  209. Zash

    ack for carbons, ack for mam, ack all teh time

  210. Ge0rG

    we need multi-ack!

  211. jonasw

    if we only had a mechanism for acknowledging stuff

  212. jonasw

    like, IQs

  213. Zash

    or send acks when the server writes into mam

  214. jonasw

    Zash, I’d like that

  215. Zash

    wait have we invented email then?

  216. lovetox

    i think thats exactly not what receipts are about

  217. lovetox

    receipt mean the client received the message

  218. lovetox

    not a server archive, which cant even determine if the message is encrypted and the client can decrypt it

  219. lovetox

    really there are error messages for when s2s or c2s does not work

  220. jonasw

    except when there aren’t

  221. Zash

    what if we just stop worrying about everything

  222. Zash

    messages get through or they don't

  223. lovetox

    yes i say this often, Gajim has receipts disabled by default

  224. lovetox

    i use xmpp now for some years, never had a problem with not received messages, except when encryption was involved

  225. Ge0rG

    .I agree with lovetox in that.

  226. jonasw

    sure, let’s not worry about reliability

  227. jonasw

    lovetox, I had

  228. Ge0rG

    lovetox: sorry, but your story is a huge anomaly. I lose messages all the time

  229. jonasw

    a lot

  230. Ge0rG

    sometimes I only lose carbons because my contact just turns off their phone's wifi

  231. Zash

    did you say something?

  232. jonasw

    but many of those would probably be solved by MAM

  233. daniel

    Ge0rG: maybe you smacks implementation is broken?

  234. daniel

    Or the fact that you don't have mam?

  235. Ge0rG

    daniel: maybe XMPP is broken?

  236. daniel

    Apparently not for me and lovetox

  237. daniel

    Who do implement both mam and 198

  238. Ge0rG

    daniel: not sure about ejabberd, but another widely-deployed XMPP server will error-bounce all stanzas when a hibernated 0198 session gets destroyed.

  239. Ge0rG

    besides, I probably just don't know zilch about how xmpp works :P

  240. daniel

    But a bounced message is not a lost one

  241. daniel

    Unless you don't track your errors

  242. Zash

    Schrödingers bouncy deliveries

  243. dos

    afaik ejabberd has a config entry for that and can either put them to offline storage or bounce

  244. daniel

    Yeah so does prosody. But complaining about things is always more fun

  245. Zash

    The day humans stop complaining is the day we gone extinct.

  246. Ge0rG

    daniel: the prosody module doesn't work if you have multiple clients

  247. Ge0rG

    But yeah, complaining is more fun.

  248. Ge0rG

    Also I'm not sure what tracking errors gives you, unless you do automatic retransmission, which is a huge PITA

  249. Maranda

    We need some ELE first or later anyways

  250. jonasw


  251. Maranda

    jonasw: Extinction Level Event

  252. Ge0rG

    Maranda: https://syonyk.blogspot.com/2015/08/i-am-backsideist.html

  253. Maranda

    sounds like some complex read which I'm a bit too tired to start :O

  254. Maranda

    > not sure about ejabberd, but another widely-deployed XMPP server will error-bounce all stanzas when a hibernated 0198 session gets destroyed. well I have the "Ge0rG's maneuver" when there's at least one session online supporting carbons :P

  255. Maranda

    (aka "assume like it's delivered")

  256. Maranda

    (... and don't bounce it)

  257. Ge0rG

    Maranda: is that upstreamed?

  258. Maranda

    Ge0rG, it's in Metronome from quite a bit yes.

  259. Ge0rG

    Maranda: that's not what I mean.

  260. Maranda

    Then I'm too tired to understand what you mean, elaborate plx :P

  261. Ge0rG

    Maranda: prosody

  262. Maranda

    And I don't think so, dunno.

  263. Maranda

    > is quite unpleasantly high, the state of infrastructure is quite sad (when was the last bridge collapse?), 6 days ago.

  264. Ge0rG

    Not speaking of irc bridges