XSF Discussion - 2020-01-20

  1. Ge0rG

    dwd: maybe an official blog post? My boss is actually rather reasonable, and I don't think there will be a convincing angle besides of that project I've told you about

  2. winfried

    Ge0rG: don't know, but maybe this can help: https://wiki.xmpp.org/web/XMPP_summits_for_dummies

  3. Ge0rG

    Probably the protocol standardization angle is even better. We are part of some such orgs, like IEC 62443

  4. Guus

    Ge0rG didn't we have a blogpost for your PHB last year? 🙂

  5. Ge0rG

    Guus: it doesn't explain anything about Summit, merely announces it

  6. nyco

    one for the iTeam? https://twitter.com/BenBE1987/status/1218983250833608704

  7. Guus

    nyco they're aware. Someone already brought it up during the weekend (technically, it's not iteam, but some of the same people are involved)

  8. nyco

    oh sorry

  9. Guus

    no worries, easy mistake

  10. Ge0rG

    I'd like to know the reasons why it's not iteam (or in other words, why jabber.org is not run by the XSF)

  11. jonas’

    I bet those reasons are hysterical

  12. Ge0rG

    IIRC that came up before and there also was something regarding separation of concerns

  13. MattJ

    Historically jabber.org and the JSF/XSF were very intertwined

  14. MattJ

    So it was an active move to separate them, not historical reasons

  15. MattJ

    I'm not sure how much sense it makes for a standards foundation to run a public IM service

  16. Ge0rG

    something something matrix.org

  17. Guus

    Does it matter either way?

  18. MattJ

    Yes, if the past few weeks of the XSF have shown anything :)

  19. Guus


  20. MattJ

    It's currently running a closed-source XMPP server

  21. Guus

    noooooo you didn't go there didn't you? 😃

  22. jonas’

    oh dear

  23. Zash

    XSF = evil confirmed

  24. MattJ

    I meant to demonstrate that it does matter, I don't believe XSF manpower should be distracted from standards work by discussing some random IM service

  25. jonas’

    but jabber.org isn’t the XSF!

  26. jonas’

    MattJ, I’m not sure that one of the largest servers which also has the name matching exactly the trademark the XSF has some power over should be called "random" ;)

  27. jonas’

    but yeah

  28. jonas’

    it needs lots of work to be good

  29. MattJ

    Obviously jabber.org has some historical significance, but it's conceptually nothing to do with standards development (more than any other public server is)

  30. Daniel

    i tried to address some of the LC feedback for 363. https://github.com/xsf/xeps/pull/880 cc Kev, Guus

  31. Guus

    seen it. I think its an improvement.

  32. Guus

    wasn't to worried about the previous state, but this makes it more concise.

  33. Daniel

    yes. you were right that it was a weird requirement (someone else pointed that out as well; but i forgot who otherwise i would have cced them as well)

  34. dwd

    Daniel, The requirement of no access control? I pointed that out. I note you didn't include that as a security consideration; it's implied, and not a hill for me to die on, but it seems worth considering if you're building a system using this spec.

  35. dwd

    Daniel, More explicitly: I don't think the lack of access control is a bad thing; I do think it's the kind of thing people should be aware of, and perhaps having it in the requirements is enough.

  36. pep.

    One can still run access control on the network. https://upload.some.lan/foo.png might not be accessible to people outside the network

  37. Daniel

    update https://gultsch.de/files/xep-0363.html

  38. Daniel

    (in section 8.3)

  39. dwd

    Daniel, Looks good - that's not in the current PR unless I'm being more stupid than usual?

  40. Daniel

    no; i wanted to run that by you before i force push

  41. dwd

    Daniel, Ah, I see. It looks fine, push, merge, etc.

  42. ralphm

    You can combine this, e.g., with pre-signed URLs, like Amazon S3 has.

  43. ralphm

    At VEON we did something like this: client requests a slot, server returns pre-signed URLs for PUT and GET. Even though there's strictly no access control, as per section 2, it is a *lot* harder to know the URLs.

  44. ralphm

    Another thing you may want to consider is having the links expire after a while. Very short for PUT, e.g. a week for GET.

  45. Daniel

    I actually have a deployment too where the get urls expire. I find that very annoying because I have to go through the service to get them renewed

  46. Daniel

    I can't just put a url in the avatar meta data node for example

  47. ralphm

    Ah, indeed, for avatars, we had something special.

  48. ralphm

    Expiry is quite useless there.

  49. ralphm

    I suppose it would be nice if you could specify desired expiry.

  50. ralphm

    (when requesting a slot)

  51. Ge0rG

    Which is also useless for avatars

  52. Ge0rG

    Because you don't know when you'll create the next one. And eternal storage is a nice DoS vector

  53. Guus

    still, end-users generally seem to expect non-expiry.

  54. Guus

    I have a simple server-sided implementation. The number one complaint was that stuff eventually disappeared (which was by design).

  55. jonas’

    expiry in the order of months is probably ok, plus a quota

  56. Guus

    jonas’ I wonder if it is.

  57. Guus

    if expiry takes that long, people might have grown to expect data to persist.

  58. jonas’

    Guus, or have forgotten about the links ;)

  59. Guus

    the amount of people that complain will be less, but those that do have an issue, will have a bigger axe to grind.

  60. jonas’

    one can of course also do something smarter than this, but I’d also like to note that in general, clients present HTTP upload as a method to transfer files, not as a method to store things

  61. Guus

    I don't think so.

  62. jonas’

    then that’s a client issue IMO

  63. jonas’

    you cannot reasonably expect an XMPP service to host files forever

  64. Guus

    people are used to be able to scroll back in history endlessly, and view media that was exchanged in a chat from other devices, pretty much unlimited.

  65. Guus

    jonas’ I think end-users do, to be honest.

  66. Guus

    they don't care about the poor system admin having to pay for storage 🙂

  67. jonas’

    Guus, I disagree ;)

  68. jonas’

    but eh

  69. jonas’

    that’s a pointless discussion then and we’ve had enough of *that*

  70. Guus

    the bikeshed shall be purple!

  71. jonas’

    let’s boil it down to this: expiry is going to be a real world thing which happens

  72. jonas’

    be it based on time, or based on quota and fifo

  73. Guus

    what we could do is make it more explicit.

  74. Guus

    managing expectations goes a long way.

  75. jonas’

    indeed, which is what I meant by saying "that’s a client issue IMO"

  76. Guus

    again, I disagree 🙂

  77. Guus

    you want your media to show up consistently on all resources.

  78. Guus

    I don't think clients can / should manage that.

  79. jonas’

    by client issue I mean: clients should be clearly communicating that HTTP upload URLs are ephemeral

  80. Guus

    ah right.

  81. jonas’

    which is I think what you were saying

  82. Guus

    yeah - I was thinking more of clients explicitly negotiating this with the server (which the xep already at least partially allows for, iirc)

  83. ralphm

    We did some calculations on what it would cost to store uploaded pictures and videos. At any significant scale, this is prohibitive.

  84. ralphm

    I.e. if you'd allow indefinite storage.

  85. ralphm

    And sure, I dislike stuff going away quite a bit, but there is so much more media exchange in typical chat exchanges than compared to, say, e-mail, that I think it is very reasonable for server admins to decide on relatively low expiry, in the order of a few weeks.

  86. ralphm

    For avatars, of course, you should be able to select "indefinite"

  87. MattJ

    What's the difference?

  88. MattJ

    Avatars are small? and you only have one of them?

  89. edhelas

    i'll have to limit en cleanup old files on my xmpp server as well

  90. ralphm

    MattJ: if avatars would expire, that be not be great

  91. MattJ

    ralphm, I agree

  92. MattJ

    But we're against indefinite hosting of stuff, and now we're making exceptions - I could upload a 6GB movie and claim it's my avatar

  93. MattJ

    Just trying to figure out what the rules should be

  94. ralphm

    MattJ: what a reasonable size is, is debatable, but it doesn't compare to typical media sharing behavior I witnessed at VEON, or in, say, WhatsApp

  95. ralphm

    MattJ: I could imagine servers offering different "buckets" for different use cases. One for avatars (you can only have one, max this size, no expiry), v.s. media sharing (many, max total size, max expiry)

  96. ralphm

    And that clients have to specify which bucket to use.

  97. ralphm

    _also_, if you are sharing stuff from an existing storage service, like a picture that is also stored in Google Photos, maybe it would be much better to share by reference.

  98. ralphm

    Then, the recipient's ability to go back and still retrieve the image is dependent on that service, instead of your server.

  99. ralphm

    Lastly, if you are concerned about your local archive, maybe servers should provide a way to copy and store media shared with you, and reference that from your MAM archive.

  100. jonas’

    ralphm, re sharing by reference, that’s already "supported" by some clients (e.g. in poezio, you can /embed any URL instead of /upload local file)

  101. dwd sets avatar as 1Tb tarfile.

  102. jonas’


  103. jonas’


  104. ralphm

    jonas’, yeah, so Gajim just shows a link, but Conversations shows the picture.

  105. ralphm

    I installed some Gajim plugins now. That helps.

  106. jonas’


  107. jonas’

    (and JabberCat will simply OOB-tag any URL-only message)

  108. dwd

    I know that there's a group that resists any suggestion the XSF should make UI/UX mandates, but I'd really like to suggest renderings for some messages so we have an agreed way to send pictures etc.

  109. ralphm

    What if we'd call them Usability Considerations.

  110. ralphm

    What if we'd call them Usability Considerations?

  111. Ge0rG

    Uability is faaaar out of the scope of the XSF

  112. Ge0rG

    Also I like the idea of mirroring shared files on the user's server, but the only problem it solves is when the remote file hoster has a shorter expiration than your MAM

  113. ralphm

    Ge0rG, sure. But I control which server I use, and by extension what that expiration is. In my case: self-hosted ⇒ infinite.

  114. Ge0rG

    ralphm: yes, I understand that. But it doesn't scale to mere mortals

  115. ralphm

    Ge0rG, as for usability being out of scope, I don't see any problem with people writing XEPs discussing usability aspects. Most people writing XEPs are not security experts, but still write Security Considerations. Anyone implementing a XEP still needs to make their own determination on security.

  116. Ge0rG

    ralphm: that message contained a sarcasm element, but I forgot to add a legacy body. Sorry.

  117. ralphm

    E.g. when specifying Reactions, you can suggest how this can be / is typically rendered, in different types of clients (GUI, text-only, text-to-speech). Or point to a document like this: https://indieweb.org/reacji

  118. ralphm

    Ge0rG: I just responded as if it wasn't, as not every casual reader would be able to detect it.

  119. Zash

    Ge0rG: Like in Matrix, with some special URI scheme with some touple of server name and file id, retrieved from your own server always?

  120. Ge0rG

    Zash: something like that. would also close the ip leak to adversarial servers

  121. Zash

    Server-hosted very strict proxy? Chained Proxy65? Magic?

  122. Ge0rG

    apt install squid

  123. ralphm

    But with a stable cache.

  124. ralphm

    That persists after the original is gone.

  125. Ge0rG

    This is a can of worms.

  126. ralphm


  127. ralphm

    E.g. when signing individual messages

  128. Ge0rG

    But somebody wanted a quick&dirty solution to file sharing.

  129. Guus

    Is there an existing implementation of MAM that allows for keyword search? If so, what's the form field definition that's used for it? XEP-0313 doesn't specify that functionality (but does list an example 'urn:example:xmpp:free-text-search')

  130. Guus

    I'm adding something for Openfire. Might as well re-use something existing.

  131. Guus

    cc MattJ

  132. Zash

    MAM that ships with Prosody doesn't have full text search

  133. Zash

    ejabberd might have?

  134. MattJ

    Guus, ejabberd has something, and I have a draft spec in my head

  135. Guus

    just need the field form definition for now 🙂

  136. MattJ

    The main issue is that you have to do dumb search or specify what operators are supported

  137. MattJ

    Dumb search is fine for some users, but advanced search in e.g. Slack is quite helpful

  138. MattJ

    But they can be separate specs obviously

  139. ralphm

    I've been contemplating learning to program a Lua module to write a MAM module for Prosody.

  140. ralphm

    That uses Elasticsearch as backend.

  141. Guus

    I'm faking stuff by feeding it in a lucene query

  142. Guus

    simple keywords will work, but more elaborate lucene queries too (although you'd need to know the index fields)

  143. MattJ

    ralphm, you'd want to implement a storage module, we have an internal API which should map fairly nicely to ElasticSearch

  144. MattJ

    Guus, that's the kind of thing I wanted to avoid

  145. ralphm

    MattJ, are you aware of anyone doing this before?

  146. MattJ

    ralphm, I don't think so... it was somewhere on my todo list but everyone I've worked with has been fine with postgres for now so I never had an excuse to implement it

  147. ralphm

    I also need to figure out a way to sync my Gajim history /back/ to MAM.

  148. Ge0rG

    can't you just expose SQL?

  149. Zash

    Free-form WHERE clause?

  150. Ge0rG

    free-form SQL

  151. Daniel


  152. Zash

    Mmmmm `UPDATE "users" SET 'admin' = 1 WHERE "username" = me`

  153. Guus

    urn:irf:xmpp:free-text-search it is!

  154. Zash


  155. Guus

    ignite realtime foundation

  156. Zash

    Have you registered that?

  157. Daniel

    Guus, have you checked with ejabberd?

  158. Guus

    4 minutes of googling didn't turn up anything of interest.

  159. Daniel

    give me a second; i know xml

  160. Daniel

    the field is called 'withtext'

  161. Zash

    Not namespaced?

  162. Daniel


  163. Zash calls the registry police

  164. Guus

    withtext works for me.

  165. Guus

    thanks Daniel

  166. MattJ

    Meh, very non-standard

  167. MattJ

    Form fields that aren't standard are meant to use Clark notation

  168. ralphm

    Yeah, don't invent your own URN namespace.

  169. MattJ

    withtext is not specified in any XEP or registrat

  170. MattJ

    withtext is not specified in any XEP or registry

  171. Zash

    MattJ, hm, does that example use clark notation? It probably should

  172. MattJ

    Maybe, but it's an example of what another XEP might define

  173. Guus

    I'm either going with compatibility with ejabberd, unless someone suggests a semi-futureproof alternative in the next few minutes.

  174. MattJ

    If it's just one implementation going rogue, fine, but if everyone follows it's kinda annoying

  175. MattJ

    Especially since your search syntax *isn't* plain text, but lucene

  176. MattJ

    so '{xmpp:openfire.org}withtext' or whatever you want

  177. ralphm

    MattJ: what does plain text mean, exactly? Every implementation needs to interpret the query string somehow. A few examples: a) does the input get tokenized? b) do the tokens get stemmed? In which language? c) does a space mean AND or OR d) any other processing like case folding

  178. Zash

    Fields can have a description. Just paste the entire docs for your fts engine in there :)

  179. Daniel

    technically it probably doesn’t matter much; you just take direct user input; and then depending on server users might see different results; but than can be ok

  180. Daniel

    i mean search doesn’t need to interop between multiple servers

  181. Guus

    it's kinda nice that you get somewhat similar results though

  182. dwd

    ralphm, What Daniel says. If you're asking for a plaintext search, then the interop point is that you can ask, not what results you get back (which is a QoI case).

  183. ralphm

    Daniel: Sure, but if you want to offer client-side assistance, like how Slack suggests search modifiers, then maybe it would be helpful to figure out how to relay that information.

  184. Guus

    Yeah, but it's annoying if you're in three different MUCs, and the search on your client yields different results based on the same syntax (eg: spaces result in an AND or OR query)

  185. Daniel

    well you could probably specificy low hanging fruit like "it means AND"; but i don’t think stemming needs to be the same across servers

  186. dwd

    Guus, How would you tell?

  187. dwd

    Guus, I mean, unless they were interlinked MUCs via FMUC or something equally esoteric.

  188. Guus

    dwd: say I'd remember having a conversation, but forgot in what room. I'd like to use the same query on all rooms.

  189. Ge0rG

    dwd: you can't, so it's even more annoying

  190. Zash

    Local MUC history?

  191. Zash


  192. Zash


  193. dwd

    Surely any server with search that didn't find stuff in any obvious way would just get fixed?

  194. Guus

    surely. I'm currently fixing the fixes applied by the previous fixer.

  195. Guus eyes dwd

  196. Guus


  197. Guus

    does this make anyone extremely unhappy?

  198. Guus

    form.addField("xmpp:irf:mam:free-text-search", "Free text search", FormField.Type.text_single);

  199. Guus

    (other than that it's Java)

  200. Zash

    Guus, that makes me unhappy

  201. Zash

    Oh, wait, not urn:

  202. Guus

    does this make anyone but Zash extremely unhappy?

  203. jonas’

    Guus, I’d prefer {urn:dns:openfire.org}free-text-search or something similar

  204. Zash

    or `{xmpp:openfire.org/stuff-maybe}free-text-search` like MattJ said earlier

  205. Daniel

    is urn:dns:mydomain a thing?

  206. jonas’

    it is

  207. Zash

    is it?

  208. Daniel


  209. Zash

    I don't see it in https://www.iana.org/assignments/urn-namespaces/urn-namespaces.xhtml

  210. jonas’

    ah, dns:mydomain

  211. jonas’

    not urn:dns:

  212. jonas’


  213. Zash

    or `urn:uuid:$(generate-uuid)` :)

  214. jonas’

    or that, but that’s quite opaque

  215. Zash

    Nice for experiments tho

  216. jonas’


  217. Zash

    I had a thing that used a hash based UUID over its own source code.

  218. MattJ

    Great for interop

  219. Zash

    Yeah, either you have the same version or it doesn't work.

  220. ralphm

    urn:dns is silly for this, I think

  221. jonas’

    urn:dns isn’t even a thihng

  222. ralphm

    Just use https URLs

  223. Zash

    No, use gopher: URLs!

  224. MattJ

    This is *definitely* the protocol developers' equivalent to bikeshedding

  225. Guus

    Yeah, I'm back at freetext.

  226. Zash

    Hold on let me find where ASN.1 namespaces are mounted into the urn: tree

  227. ralphm

    Yes, indeed, but using a non-registered URN namespace is just *wrong*.

  228. Guus

    or whittext or whatever it was that ejabberd did

  229. Zash


  230. MattJ

    Guus, you've had a number of acceptable suggestions, choose one or state why they aren't good enough

  231. MattJ

    xmpp:irf isn't acceptable because 'irf' isn't the address of an XMPP service

  232. Zash

    Unless you register the .irf TLD for 200k :)

  233. ralphm

    And then still, single-label DNS names are, well, terrible.

  234. Guus

    Yeah, I'm going for the same as ejabberd until there's consensus.

  235. MattJ

    > 15:36:49 Zash> or `{xmpp:openfire.org/stuff-maybe}free-text-search` like MattJ said earlier

  236. ralphm

    Guus: what does ejabberd do?

  237. MattJ

    There won't be consensus because almost all the suggestions were valid options

  238. MattJ

    All more valid than 'withtext'

  239. Guus

    any client that currently supports the feature will probably use 'withtext'

  240. Guus

    ugly as hell, but at least that adds some functional value.

  241. Zash

    And thus we have another de-facto standard.

  242. MattJ

    With a different syntax

  243. MattJ


  244. Zash


  245. Ge0rG

    I suggest the eu.siacs.conversations.withtext namespace.

  246. ralphm


  247. MattJ

    Guus, to be clear, it is XEP-0068 you are breaking by using 'withtext': https://xmpp.org/extensions/xep-0068.html#approach-fieldnames

  248. MattJ

    (and so is ejabberd, but it's not too late to save you)

  249. Guus

    I'm happy to change it whenever there's consensus on something - a protoxep, or whatever. I've now had various suggestions that were all more or less desirable - I simply don't have the time to evaluate all options today.

  250. ralphm

    Guus: Indeed, please just use clark notation with an https URI.

  251. Guus

    stuff hasn't been released yet anyway

  252. MattJ

    Guus, the problem with protocols is once you do that, you're stuck with them... need I mention vcard-temp?

  253. ralphm

    Guus: it is more important to not do the things people said are definitely wrong. Including 'withtext' (unqualified), unregistered URN prefixes, stuff starting with 'urn:xmpp:' when the doc is not in our inbox.

  254. ralphm

    MattJ: or even jabber:

  255. Zash

    > need I mention vcard-temp? WHY WON'T IT DIE? I've been trying to push for vcard4 but everyone just uses more vcard-temp, like adding it to MUC :(

  256. ralphm


  257. Zash

    But then XEP-0084 is sorta supported by everyone and Pidgin