XSF Discussion - 2018-03-06

  1. Ge0rG

    pep.: there is an alternative development in https://github.com/ge0rg/easy-xmpp-invitation/tree/bootstrapless

  2. Ge0rG

    pep.: unfortunately, Martin and me did some work in parallel, because I forgot to push 🤦

  3. Zash

    What hip new framework is that? Is it anything like vanillajs?

  4. jonasw

    Ge0rG, did you mean to ping marc?

  5. Ge0rG

    jonasw: right, I must've been scrolling too fast.

  6. Ge0rG

    Sorry pep., I meant to respond to marc indeed.

  7. Zash


  8. pep.

    Ge0rG: thanks for the info ;)

  9. Ge0rG

    jonasw: is there a tool to maintain xeps/xep.ent? I'm missing &xep0401; for an update to PARS.

  10. jonasw

    Ge0rG, not yet

  11. jonasw

    I can either update it right now or you can do it yourself

  12. jonasw

    please pick one

  13. Ge0rG

    jonasw: please do so, thanks :)

  14. jonasw

    Ge0rG, I guess I don’t need to botehr with issueing a deferral for XEP-0379 (PARS) then? ;-)

  15. jonasw

    Ge0rG, xep.ent updates pushed

  16. Ge0rG

    jonasw: thanks very much. I have had some pending minor changes for 0379 that I wanted to push before mid-Feb.

  17. Ge0rG

    Wanted and failed to.

  18. jonasw

    lets keep the spam low and defer the deferral for now then

  19. Ge0rG


  20. jonasw

    tbh it’s more laziness than anything else ;-)

  21. jonasw

    that and trying to prevent my domain from getting a bad spam score for all those mails ;-)

  22. Ge0rG

    Would it make sense to have the same xmlns for 0379 and 0401?

  23. Ge0rG

    0401: <preauth xmlns='urn:xmpp:invite:1'/> 0379: <preauth xmlns='urn:xmpp:pars:0' token='1tMFqYDdKhfe2pwp' />

  24. jonasw

    only if you want to tie into namespace bumps of either

  25. jonasw

    otherwise I’d suggest not to

  26. jonasw

    but I lack context

  27. Ge0rG

    Right. That might be unfortunate.

  28. Ge0rG

    Though I'd like to burn the notion of namespace bumps with fire anyway.

  29. jonasw

    what’s your suggestion what we should be doing instead?

  30. Ge0rG

    jonasw: feature-flags

  31. Zash

    and the difference is?

  32. jonasw

    hm, those help in some cases, but they don’t always make sense

  33. Ge0rG

    Zash: the difference is that it's a better protocol design and that it won't break people's expectations when they upgrade conversations without also upgrading ejabberd.

  34. jonasw

    Ge0rG, how?

  35. jonasw

    the http upload update is a particularly bad example. it changed the whole syntax of the thing

  36. jonasw

    even *if* that was represented by a feature, you’d still most likely implement only one variant.

  37. Ge0rG


  38. Ge0rG

    Maybe I haven't thought through this.

  39. Zash

    It's less terrible once the protocol becomes more stable

  40. jonasw

    it helps with small things; if there had not been a syntax change but merely the addition of support for http headers, sure, that would work fine as a feature. however, the server would also need to detect that the client supports that. requires either caps, an additional roundtrip or a payload in the request.

  41. Zash

    More options doesn't equal better

  42. Ge0rG

    jonasw: https://github.com/xsf/xeps/pull/599

  43. jonasw

    I don’t consider the URI change editorial fWIW

  44. jonasw

    but that doesn’t matter for you

  45. Ge0rG

    I don't even like the URI change.

  46. jonasw

    (it *does* matter in the sense that this would otherwise have been the trigger for a discussion on whether purely editorial changes should delay deferral)

  47. Ge0rG

    I could be asking for a Last Call instead, but somehow nobody implemented PARS in the last year.

  48. jonasw

    I’m sorry

  49. jonasw

    (I actually am)

  50. Ge0rG

    I am as well, because I considered this a very effective way to make Jabber user onboarding easier.

  51. Ge0rG

    Unfortunately, some other widely-used implementation's author objected on formal grounds.

  52. jonasw


  53. Ge0rG

    Oh, in unrelated news: it seems like yaxim has over 100k total downloads on Google Play. I'm impressed

  54. jonasw


  55. jonasw

    (that’s like gz, but different)

  56. Ge0rG

    So different I'm not getting it. Sorry.

  57. Zash


  58. jonasw

    Ge0rG, man lzop

  59. Ge0rG

    jonasw: so while I'm impressed, you are compressed?

  60. jonasw

    Ge0rG, oh dear. so, gz is a short form of a leety "congratz" from back in the olden days when that was cool. or people thought it was cool.

  61. jonasw

    does that help?

  62. Ge0rG

    jonasw: ah, now I see the arc you are bending. Thanks.

  63. jonasw

    docker hub has the worst web UI ever.

  64. jonasw

    i have no idea why I need to "cancel" a build like three times before it takes effect. the first time the UI just won’t respond at all, the second it’ll change the button to "cancelling" but nothing further happens (after a reload it’s back to normal), the third one is finally effective

  65. jonasw

    not to mention the "X minutes ago" timestamps which only refresh on areload

  66. jonasw

    but which somehow suggest that the view was live

  67. jonasw


  68. Zash

    Pray to the Browser overlords for magic no-js fuzzy styling of <time> tags

  69. jonasw

    Kev, wouldn’t a nicer way to do that "MAM <before/>+<after/>" be a form field in the query?

  70. lovetox

    it not only nicer, anything else would be in every way wrong

  71. Zash

    But paging is defined by RSM.

  72. jonasw

    Zash, that’s exactly why I’d prefer to have this in the form. alternatively, define the direction in that case via the form.

  73. jonasw

    and update RSM accordingly

  74. jonasw

    lovetox, regarding "start to sync again from that ID", that might be arbitrarily inefficient.

  75. lovetox

    we already concluded that mam is sometimes inefficient

  76. jonasw

    no need to make it worse :)

  77. lovetox

    RSM has nothing to do with what you want to do

  78. jonasw

    lovetox, huh;

  79. lovetox

    you want to query archive-id 😶 to archive-id = Y

  80. jonasw

    there’s an emoji in your message

  81. lovetox

    that should be defined in the query to MAM, like startime and endtime

  82. jonasw

    that sounds reasonable, but everyone is just talking about using <before/> and <after/> for that

  83. lovetox

    RSM is only for browsing the result

  84. lovetox

    that makes no sense

  85. Zash

    FWIW, in Prosody, direction is set to backwards if <before/> is present

  86. Zash

    What happens after that depends on the lower storage layer

  87. MattJ

    Zash, is that even standard?

  88. lovetox

    <before/> gives you the last page of a result

  89. lovetox

    that is standard

  90. Zash

    MattJ: What, exactly, are you asking?

  91. MattJ

    Zash, afair in XEP-0313 results are always transmitted chronologically

  92. jonasw votes to close MattJs question as "Unclear what you’re asking"

  93. jonasw

    (scnr and just kidding)

  94. Zash

    MattJ: That's taken care of. This is different.

  95. Zash

    So you should get a page where the last item preceeds the one identified by <before>

  96. Zash

    If you set <after> ... then either the results include that item or they don't

  97. Maranda votes for "they don't"

  98. Zash

    There's no voting

  99. Maranda


  100. jonasw

    Zash, tell that to council

  101. intosi

    Zash: FWIW, in M-Link direction is set to backwards if <before/> is present, as well.

  102. intosi

    And specifying after never includes that item in the result set, regardless of whether you used before or not.

  103. Maranda

    I think that's how most implementations would do anyways? Is there any "black sheep" not doing that 🤔?

  104. Maranda

    (about <before /> going backward from x)

  105. intosi

    MattJ: the way I read RSM, that's how it should behave. before/ is quite explicit about directionality :)

  106. Zash

    I'm not promising that all Prosodys storage backends deal with the value of <after> if <before> is also present

  107. Kev

    > jonasw > Kev, wouldn’t a nicer way to do that "MAM <before/>+<after/>" be a form field in the query? Why? RSM's already defined.

  108. Zash

    Does RSM define what happens if you include both?

  109. intosi

    Zash: it does not. But it similarly doesn't tell you you can't.

  110. jonasw

    Kev, I checked the other day and didn’t find a definition of what happens both <before/> and <after/> are given.

  111. Kev

    Zash: No, but if we're just careful in 313 that we explain it, we're golden.

  112. Kev

    I'm not suggesting that no text tweak to 313 is needed. Just that we already have the protocol and don't need to invent more.

  113. Maranda just checked, he doesn't handle a case if both are present.

  114. Maranda


  115. jonasw

    Kev, I’d argue anything which specifies RSM behaviour should not be in 313, but in the RSM XEP.

  116. Kev

    jonasw: I think that, in this case, I'd disagree.

  117. Kev

    But equally I wouldn't really mind it being in 59 either.

  118. Zash

    Default file based storage doesn't support both, so results would include the item in <after>.

  119. Maranda

    <before /><after>uuid</after> glitches a bit in my mind <before>uuid</before><after>uuid</after> sounds like in between

  120. Maranda

    And glitches too

  121. intosi

    Maranda: your first example asks for the last items according to your requested page size, or some server default (might be inifity), but nothing that's uuid or older.

  122. intosi

    Note that servers are very likely not to allow infinity-sized pages, as that's quite a burden on resources ;)

  123. Ge0rG

    Is the RSM discussion led in the context of "Dusty Drafts"? :D

  124. Ge0rG

    intosi: I had to learn the hard way, that even with a limited page size a DB might still lock up the whole server for multiple minutes merely to count the total number of results.

  125. Zash

    O(n) all the things

  126. Ge0rG

    Mar 06 14:28:30 mod_log_slow_events warn Slow event 'iq-set/self/urn:xmpp:mam:2:query' took 12.01s: ...

  127. intosi

    YEs. Counting items is a terrible idea.

  128. Ge0rG

    intosi: except it's required by RSM

  129. intosi

    Just as expensive as generating the entire set, usually.

  130. intosi

    Ge0rG: read closely.

  131. Holger

    > <jonasw> Kev, I'd argue anything which specifies RSM behaviour should not be in 313, but in the RSM XEP. +1. I don't want an RSM implementation and a separate MAM-RSM variant.

  132. intosi

    Obtaining the item count is, but including the number of items in 313 is weasleworded.

  133. intosi

    Obtaining the item count with RSM techniques is

  134. Ge0rG

    intosi: hm. "The responding entity then returns the item count, which MAY be approximate rather than precise if determining the exact number of items would be resource-intensive"

  135. Zash


  136. Holger

    Just always return 42.

  137. intosi

    No, return requested+1 if you know there is more.

  138. intosi


  139. Zash

    intosi: Isn't that implied by complete=false

  140. intosi


  141. Ge0rG

    In SQLite, adding a `LIMIT` clause will change the order, unless you have an explicit `ORDER BY` in which case it's still going through the whole list.

  142. intosi

    That indicates that between the items specified, you may not have returned all items you think there could be in the set.

  143. intosi

    Ge0rG: depends on which data set you order-by.

  144. intosi

    Query optimisation is an art more than a science.

  145. Ge0rG

    intosi: Unforuntately it is. Especially if you have abstracted away DB access through some generic library.

  146. intosi

    Simple. Never do such a silly thing, unless you own the abstraction as well ;)

  147. Ge0rG

    intosi: tell that Zash :P

  148. intosi

    Zash: what I said^

  149. Zash

    intosi: Tell Ge0rG that SQL things are terrible and why we can't have nice things

  150. Ge0rG

    intosi: and don't tell me not to use software where I don't control the whole stack, down to CPU µcode.

  151. intosi


  152. Ge0rG

    Because I'd rather spend my time on Easy XMPP

  153. Ge0rG

    I mean Jabber.

  154. Maranda

    well counting isn't an issue here (as I never operate on the database directly), yet since at the time reading RSM spec I thought a before+after combo wasn't a possibility I don't handle those cases.

  155. Ge0rG

    I mean Zimpies.

  156. Maranda

    Metronome will handle before but not after if both are present.

  157. jonasw

    Maranda, at least return an error when you can’t handle a case and know it :<

  158. Zash

    Ge0rG: How much do you care about the total anyways?

  159. Maranda

    jonasw, I'll add something to handle those now.

  160. Ge0rG

    Zash: I care about my server not lagging to death.

  161. Ge0rG

    Zash: no idea how much MAM-enabled clients rely on the `count` value

  162. jonasw

    Zash, just exploit 313 and return number-of-items+1

  163. Holger

    Maranda: Yes, I think it's quite obvious that the idea behind 0059 is to specify *either* <before/> *or* <after/>, and this is just not made explicit because the author didn't even consider the possibility to specify both.

  164. Zash

    If you disable archive pruning, it stops counting.

  165. Holger

    So I think we can't just cheat and allow both in 0313.

  166. Holger

    (Though I clearly see the use-case of course.)

  167. jonasw

    lovetox suggested to use form fields instead of <before/>+<after/>, which I find sane

  168. Kev

    Holger: I don't think it's cheating, and we clearly can, because we have :)

  169. Holger

    jonasw: Yes, me too.

  170. Zash

    You break a MUST in 59 if you include both and the range is larger than one page

  171. Zash

    > first item [...] MUST be the item that immediately follows [...] in the <after/>

  172. Holger

    Kev: Well yes we can do ugly things, and I agree we have.

  173. intosi

    Zash: well spotted.

  174. jonasw

    Zash, only if you iterate backwards, or?

  175. Zash

    > last item in the page [...] MUST be the item that immediately preceeds [...]

  176. Ge0rG

    Zash: we can override a MUST with follow-up spec

  177. jonasw


  178. Kev

    But doing before/after in 313 is possible. It's just going to get ugly specifying the same things twice.

  179. jonasw

    Ge0rG, but then clients need an implementation of RSM and one of MAM-RSM

  180. jonasw


  181. Kev

    jonasw: They do anyway, in practical terms.

  182. Ge0rG

    jonasw: hm.

  183. jonasw

    Kev, do they?

  184. Holger

    jonasw: And servers.

  185. intosi

    RSM is very protocol specific anyway.

  186. Holger

    Kev: Why?

  187. Kev

    Sorry, not clients, servers. Clients don't.

  188. Holger


  189. jonasw

    I have an interface in mind which is completely agnostic to protocol which uses RSM by passing a function around.

  190. Zash

    jonasw: next?

  191. jonasw


  192. jonasw

    where RSM would just be a building block to obtain an asynchronous iterator

  193. Zash

    jonasw: If I were building something today, I'd make it more like a file handle

  194. MattJ

    Requiring server-side state?

  195. jonasw

    and I’d like to keep it that way.

  196. jonasw

    server side state sounds awful

  197. intosi

    It is.

  198. MattJ

    It would be, yes

  199. jonasw

    (encode the state in an opaque base64 thing!)

  200. Zash

    MattJ: I have had the thought not limit ranges, and stash the iterator along with the current id, so one can resume from there :)

  201. Maranda

    and intosi, tbh why not just doing <before>uuid-that-was-after-then</before>...?

  202. intosi

    Cursors are not something you'd want to keep around for every query sent to your xmpp server :)

  203. Maranda

    intosi, that's what glitches

  204. Zash

    LRU cache or whatever

  205. Zash

    On the assumption that the client will likely ask for the next page right after the current, it might be worth it

  206. intosi

    Maranda: I don't understand your question, my head's only partially working.

  207. Maranda

    mine as well tbh lol, but <before/><after>uuid</after> sounds exactly like <before>uuid</before> in my head.

  208. intosi


  209. Zash


  210. intosi

    <before/> is very specific: it's before EOF, which is something that doesn't have an UUID.

  211. intosi

    You can't specify before last uuid you know, because it will never return the last uuid you know, which may not even be the last uuid in the dataset in the first place.

  212. Maranda

    Hm ok.

  213. lovetox

    you try bend rsm to do what a simple query filter does a lot better

  214. lovetox

    if i can query MAM with end and start archive ids, and my result only contains whats in between

  215. lovetox

    i can use RSM to page forward backwards wahtever

  216. intosi

    That's exactly what RSM does.

  217. lovetox


  218. intosi


  219. jonasw

    no :)

  220. jonasw

    well, kinda

  221. lovetox

    RSM pages inside a Result

  222. jonasw


  223. Zash

    <{dataform}x><field name=sql><value>SELECT * FROM archive WHERE whatever<///>

  224. lovetox

    it does not filter a result

  225. jonasw

    Zash, good idea

  226. intosi

    Zash: expecting your protoxep before CoB :)

  227. lovetox

    the Result of the server is the book, and with RSM you switch to a page

  228. lovetox

    what you try to do is to rip site 1-3 out of the result with rsm

  229. lovetox

    and create a new book

  230. Kev

    No, it's really not.

  231. intosi

    RSM is specific for a protocol, and we're talking 313, which clearly defines the uses of RSM.

  232. Zash

    lovetox: Hmm, epub over xmpp!

  233. Holger

    The part lovetox explained is not specific.

  234. Holger

    It's exactly the generic part specified in 0059.

  235. Holger

    If it's all specific then ditch 0059.

  236. Maranda


  237. Maranda

    well, until I understand clearly what to do with both and my brain doesn't kernel panic trying to I just think I'll throw an error.

  238. Zash

    Throw it all out, define the archive as a giant XML blob and do binary serialized xml delta transmission and reinvent wave?

  239. intosi

    We should've gone with IMAP from the start.

  240. Holger

    I really dislike how we keep bending/breaking our specs in other specs.

  241. Zash

    Holger: Myeah. Consistency is nice.

  242. lovetox

    RSM does a simple thing, and it does it nice

  243. lovetox

    if you want to have more elaborate querys for mam, just define fields for it

  244. lovetox

    like start-id, end-id

  245. Kev

    lovetox: For queries, yes.

  246. Kev

    But for getting pages from results, RSM is good at that.

  247. Kev

    It's just a shame that dealing with before/after isn't well specified.

  248. lovetox

    maybe i dont get it, i request a query from MAM with start-id, end-id

  249. Zash

    Is there anything preventing us from clarifying that?

  250. lovetox

    what exactly would i need before and after in the RSM page query afterwards?

  251. Zash

    Given multiple implementations doing something, sometimes, with it

  252. intosi

    Zash: clarifying sounds like a good idea.

  253. Holger

    Kev: It's not well-specified because it goes against the 0059 model. You can't take your result book and page forward from page 15 *and* backward from page 30 at the same time.

  254. Maranda

    jonasw, *ducks* actually if before and after are currently present I *already* throw an error.

  255. Holger

    Kev: Right now you specify a page and a direction. With your change, before/after are no longer directions but boundaries.

  256. Maranda


  257. Maranda

    jonasw, both of 'em that is.

  258. Zash

    Presence or lack of <before/> sets the direction atm

  259. lovetox

    exactly, and boundaries you specify in the MAM query, and not in the paging mechanism afterwards

  260. Zash

    So I think you can read it as <before>+<after> = backwards but stop at <after>

  261. Holger

    Kev: And you sound as if it was obvious that specifying both boundaries should set direction=before. This sounds is totally arbitrary to me.

  262. Holger

    Zash: Ugh.

  263. Maranda


  264. Zash

    Or does it explicitly say 'forward' if <after>?

  265. intosi

    Only before tells you anything about direction, IIRC.

  266. Kev

    We could ditch RSM from 313 completely and just go for something with the same semantics but different syntax, it just seems wrong to me.

  267. lovetox


  268. Zash

    Oh please no

  269. Zash

    Isn't MAM done yet?

  270. Maranda

    Are we about to go back to archiving..? 😋😘 *troll intended*

  271. lovetox


  272. Ge0rG

    jonasw: thanks once again for your editorial work today

  273. jonasw