XSF Discussion - 2021-02-11

  1. pasdesushi has left

  2. pasdesushi has joined

  3. intosi has left

  4. marek has left

  5. marek has joined

  6. pasdesushi has left

  7. andrey.g has joined

  8. Seve has left

  9. intosi has joined

  10. Andrzej has joined

  11. stpeter has joined

  12. stpeter has left

  13. intosi has left

  14. Andrzej has left

  15. intosi has joined

  16. intosi has left

  17. stpeter has joined

  18. stpeter has left

  19. govanify has left

  20. govanify has joined

  21. intosi has joined

  22. govanify has left

  23. govanify has joined

  24. marek has left

  25. marek has joined

  26. jcbrand has left

  27. Seve has joined

  28. stpeter has joined

  29. stpeter has left

  30. govanify has left

  31. govanify has joined

  32. intosi has left

  33. govanify has left

  34. govanify has joined

  35. Andrzej has joined

  36. stpeter has joined

  37. stpeter has left

  38. intosi has joined

  39. alameyo has left

  40. eevvoor has left

  41. eevvoor has joined

  42. wladmis has joined

  43. intosi has left

  44. andrey.g has left

  45. Andrzej has left

  46. alameyo has joined

  47. intosi has joined

  48. Andrzej has joined

  49. govanify has left

  50. govanify has joined

  51. intosi has left

  52. intosi has joined

  53. stpeter has joined

  54. stpeter has left

  55. stp has left

  56. Seve has left

  57. stp has joined

  58. chronosx88 has joined

  59. intosi has left

  60. stp has left

  61. stp has joined

  62. murabito has left

  63. murabito has joined

  64. intosi has joined

  65. stp has left

  66. stpeter has joined

  67. stpeter has left

  68. govanify has left

  69. govanify has joined

  70. intosi has left

  71. intosi has joined

  72. chronosx88 has left

  73. chronosx88 has joined

  74. govanify has left

  75. govanify has joined

  76. stpeter has joined

  77. stpeter has left

  78. Yagiza has joined

  79. serge90 has left

  80. intosi has left

  81. serge90 has joined

  82. serge90 has left

  83. serge90 has joined

  84. govanify has left

  85. govanify has joined

  86. Tobias has joined

  87. intosi has joined

  88. lskdjf has left

  89. stpeter has joined

  90. stpeter has left

  91. Seve has joined

  92. intosi has left

  93. andy has joined

  94. emus has joined

  95. intosi has joined

  96. antranigv has left

  97. antranigv has joined

  98. antranigv has left

  99. Neustradamus has left

  100. Neustradamus has joined

  101. antranigv has joined

  102. stpeter has joined

  103. stpeter has left

  104. Neustradamus has left

  105. intosi has left

  106. goffi has joined

  107. govanify has left

  108. govanify has joined

  109. govanify has left

  110. govanify has joined

  111. wurstsalat has joined

  112. intosi has joined

  113. marek has left

  114. marek has joined

  115. goffi has left

  116. govanify has left

  117. govanify has joined

  118. marek has left

  119. stpeter has joined

  120. stpeter has left

  121. marek has joined

  122. intosi has left

  123. Daniel has joined

  124. jcbrand has joined

  125. intosi has joined

  126. murabito has left

  127. mathijs has left

  128. mathijs has joined

  129. neshtaxmpp has left

  130. ti_gj06 has joined

  131. intosi has left

  132. stpeter has joined

  133. stpeter has left

  134. murabito has joined

  135. intosi has joined

  136. Andrzej has left

  137. nyco has left

  138. Guus has joined

  139. nyco has joined

  140. Guus has left

  141. Andrzej has joined

  142. LNJ has joined

  143. stpeter has joined

  144. stpeter has left

  145. goffi has joined

  146. Steve Kille has left

  147. goffi has left

  148. goffi has joined

  149. chronosx88 has left

  150. chronosx88 has joined

  151. flow

    emus, check https://wiki.xmpp.org/web/Kevin_Smith_Application_2020

  152. Steve Kille has joined

  153. LNJ has left

  154. LNJ has joined

  155. neshtaxmpp has joined

  156. Alex has left

  157. Tim has joined

  158. Andrzej has left

  159. matkor has joined

  160. floretta has left

  161. debacle has joined

  162. chronosx88 has left

  163. chronosx88 has joined

  164. deuill has joined

  165. Kev

    Yeah, just get the Board member to mail me at my isode account please.

  166. Kev

    I thought all of Board knew how to contact me by now, sorry.

  167. stpeter has joined

  168. stpeter has left

  169. alameyo has left

  170. emus

    Thanks, yeah I just checked your page

  171. Alex has joined

  172. alameyo has joined

  173. Andrzej has joined

  174. stpeter has joined

  175. stpeter has left

  176. alameyo has left

  177. arc

    goffi: when you're around, I'd love to talk about xep-0355

  178. goffi

    arc: hi, I'm here

  179. arc

    When you drafted this, did you consider using xpath instead of namespace plus attributes?

  180. govanify has left

  181. govanify has joined

  182. arc

    One thing that you have written here that I had not considered is client defined services. It is an interesting idea, though I am curious how you envision this being implemented in the real world

  183. goffi

    initially there was only namespace, the attribute has been added because it was the only to distinguish MAM for chat from MAM for Pubsub. XPath would probably complicate (and servers would need to have a handy xpath implementation available).

  184. goffi

    Note that I would love to have xpath or at least simplified xpath in XMPP

  185. arc

    XPath is certainly more complicated. But is also more versatile. One of the problems with XPath is modern implementations are rare and hard to come by.

  186. Kev

    As a standards body we’re generally wary of xpath because of the burden it places on implementations.

  187. MattJ

    FWIW XPath is also being discussed as a solution for improving push notifications currently

  188. MattJ

    Maybe 2021 is the year to embrace XPath :)

  189. Zash

    Is "simplified xpath" a thing?

  190. MattJ


  191. goffi

    I've written this years ago, so I need to refresh a bit my memory ^^. But the idea was basically that you if you want e.g. advanced PEP and your server doesn't offer that, you could have a third party implementation and ask yourself your server to redirect stanza there, without having to wait for admin or anything.

  192. Kev

    MattJ: Ah, interesting. Where’s the push notification discussion happening? I missed that if it was on list.

  193. nyco has left

  194. goffi

    Zash: I don't know if it's already a thing, but just keeping the base of path and attribute matching, without all the methods could be useful and easy to implement.

  195. Kev

    I’m coming to the conclusion that we need two distinct models for push notifications - one for e2e and one for non-e2e.

  196. Kev

    And trying to address both with the same mechanism is likely to be painful.

  197. Zash

    goffi, so just a path like `/{xmlns}name/...`?

  198. goffi

    Zash: for instance yes, maybe with attribute matching. I don't know if this would be enough for our needs though.

  199. arc

    Nobody wants to work on lower level XML software. It's not sexy. It is easily overlooked as a resume filler. Nobody will pay for it. So that boils down to students or retired developers who love XML and take it on as a challenge.

  200. Kev

    Teensy bit of hyperbole there :)

  201. Zash

    No true Scotsman wants to work on XML! /s

  202. Kev

    Zash: :D

  203. nyco has joined

  204. arc

    Kev: is it? I mean certainly there are /some/ people working on it.

  205. Andrzej

    Kev, there was noting on the list about push notifications and xpath, but it was just an idea to allow client select what it wants to have in the push notifications payload (assuming it is encrypted) https://github.com/tigase/tigase-xeps/issues/4

  206. Andrzej

    or just filter what is expected to send push notification and what is not

  207. andrey.g has joined

  208. Zash

    Including a stanza "skeleton" was suggested at some point as well, ie stripping all content and attributes, leaving only `<name xmlns="">` of each tag.

  209. Zash

    Uh, with my server dev hat on, plz no XSLT in the server

  210. Andrzej

    Zash, that was just an idea, something else allowing to get some data out of stanza and trigger push notifications would be good as well

  211. alameyo has joined

  212. stpeter has joined

  213. stpeter has left

  214. arc

    I am certainly on board for discussing a limited XDM for use with a XMPP stream

  215. Ge0rG

    I think the idea of stanza skeletons was that the skeleton gives the push server all required info for deciding whether to make a silent or a noisy push notification, without leaking any actual user data

  216. Ge0rG

    So this is a different use case from "send encrypted message payload over push"

  217. arc

    Not sure I follow

  218. Zash

    Right, giving the client everything it needs vs giving the push gateway everything it needs.

  219. Andrzej

    after online FOSDEM and Matrix usage I'm pretty sure that usage of Push & Fetch is a bad thing compared to pushing encrypted data (almost 30% of battery used by Element fetching data)

  220. Andrzej

    I think that giving control to the client would allow them to evolve quicker with less changes on the server side

  221. alameyo has left

  222. floretta has joined

  223. Ge0rG

    Andrzej: I'm not sure if FCM and APNS will be okay with carrying all your encrypted payloads.

  224. stp has joined

  225. Andrzej

    Why not? there is a limit of 4KB but it would work

  226. Ge0rG

    That said, what you fundamentally want in that case is to initiate/terminate the xmpp client session on your app server and have a custom protocol that pushes over APNS to your client

  227. Ge0rG

    because then you can optimize everything

  228. Ge0rG

    As long as you have the xmpp client connection terminated on the mobile device, it will have to reconnect to the server and to resume the 0198 session rather often

  229. Andrzej

    every 30-120s on iOS and most likely will be killed anyway

  230. Ge0rG

    Andrzej: only if you receive something that warrants an ack

  231. Andrzej

    that is why I prefer to have offline client and pushes with notifications on which user can act (ie. open app)

  232. govanify has left

  233. Ge0rG

    "Your chat program received an XML stanza. Open app to see if it was something you care about"

  234. govanify has joined

  235. Andrzej

    as for APNS and encrypted push, they've created even example for doing that https://developer.apple.com/documentation/usernotifications/modifying_content_in_newly_delivered_notifications see Listing 1

  236. Andrzej

    Ge0rG, I was usable "content" in the notification and with encryption and XPath that could work quite well

  237. Ge0rG

    Andrzej: well, I suppose you could use that to send an encrypted blob to the app

  238. Andrzej

    Ge0rG, I want usable "content" in the notification and with encryption and XPath that could work quite well

  239. pasdesushi has joined

  240. Andrzej

    I'm already doing that in Tigase & Siskin and AFAIR in Prosody

  241. Ge0rG

    Andrzej: so what you want from XEP-0357 is to pass the full stanza to the app server?

  242. Andrzej


  243. Zash

    Pass the parts you're interested in, encrypted, via the app server to the client?

  244. Andrzej

    to allow client send "XSLT" to tranform stanza in "encrypted payload" opaque to the app servers

  245. Ge0rG

    Andrzej: but the app server is under your control, so you can do arbitrary modifications ther

  246. Andrzej

    this transformation and encryption is done on the XMPP server side

  247. Ge0rG

    Andrzej: but the app server is under your control, so you can do arbitrary modifications there

  248. Andrzej

    yes, but XMPP server is leaking user data to the app server in your example

  249. Ge0rG

    Andrzej: that's a very significant effort for the xmpp server. What key should it use?

  250. Ge0rG

    also that means the xmpp server needs to have significant knowledge over the used client / app-server infrastructure

  251. Andrzej

    client with transformation would upload key, in my case AES128 key

  252. Ge0rG

    Why not an ec25519 key?

  253. Ge0rG

    You don't really need the server to be able to decrypt the payload, right ;)

  254. Andrzej

    right, but it is aware of it anyway, so it can decrypt it

  255. Ge0rG

    I am not sure if the trade-off of leaking the private data to the app server is a big thing.

  256. Andrzej

    I'm not sure which algorithm is faster, I've assumed AES128 is good in this case

  257. Ge0rG

    Well, encryption isn't the bottleneck; XSLT is

  258. pasdesushi has left

  259. Zash

    XSLT seems like overkill.

  260. Andrzej

    As I've said, it does not have to be XSLT, but could be list of fields + XPath to fetch then

  261. Andrzej

    As I've said, it does not have to be XSLT, but could be list of fields + XPath to fetch them

  262. Kev

    > Uh, with my server dev hat on, plz no XSLT in the server M-Link has xslt to multiple reasons :| > just an idea to allow client select what it wants to have in the push notifications That seems sane at first thought.

  263. arc

    Andrzej: I tend to use XPath as a generic term for XDM myself so this is no way a criticism, I'm just wondering if you are talking about XPath proper

  264. pasdesushi has joined

  265. Andrzej

    I'm not sure, I was thinking (as XSLT cannot be used) to create some XEP that would specify filteiring data and fetching them from existing XMPP schema in to some structure, basically transforming stanza into something else

  266. Andrzej

    to filter and fetch data I was thinking about using XPath, but I could be wrong

  267. pasdesushi has left

  268. pasdesushi has joined

  269. Ge0rG

    Andrzej: what about a whitelist of xml namespaces to retain in the <message> element? would that suffice?

  270. Ge0rG

    something like https://hg.prosody.im/prosody-modules/file/64b7daa6c42c/mod_csi_battery_saver/mod_csi_battery_saver.lua#l105 but more formalized

  271. Andrzej

    as we have 4KB hard limit, I think that might be not enough to just filter elements

  272. Andrzej

    but if clients could decide what should be included that would be a step forward

  273. Zash

    Namespaces does seem like a plausible Good Enough, and is easy to implement.

  274. jonas’

    4kb after base64, right?

  275. Andrzej

    yes, correct, after base64

  276. jonas’

    so in effect just about 1kB

  277. Andrzej

    not 3KB?

  278. Zash


  279. Zash

    Depending on how much we trust the app server to not be /too/ evil, we could stuff some compression in there too

  280. Andrzej

    Zash, I think this could be too big and in some cases it is more valuable to lose some of the message <body/> but deliver notification correctly

  281. pasdesushi has left

  282. Zash

    How big are your messages?

  283. arc

    Xmpp does not implement full XML. Never did. It is a subset. So I don't see a problem with creating a subset of XDM

  284. Ge0rG

    So you end up encoding the limits of google and apple services into my server.

  285. arc

    XPath and xquery both use XDM

  286. jonas’

    right, 3kB, not just 1

  287. jonas’

    Zash: with omemo, messages can get big quickly

  288. Zash

    I was just thinking that, with omemo, the server can't do anything like ship half the <body>

  289. Andrzej

    is server is aware of clients OMEMO device id it could filter out keys in the notification and notification could still work

  290. jonas’

    XDM seems to be xpath 2.0, which afair lacks implentations and is may be overkill

  291. Andrzej

    if server is aware of clients OMEMO device id it could filter out keys in the notification and notification could still work

  292. floretta has left

  293. Wojtek has joined

  294. arc

    Referring back to my earlier statement about how nobody wants to work on XML stuff 😆

  295. Zash

    You can have all of XPath as long as it's only `/{xmlns}name/{otherns}foo@bar/` and nothing else.

  296. mathieui

    Zash, and attribute matching!

  297. Zash


  298. arc

    At this point we're really talking about XPath 3.0 or quite possibly XPath 4.0

  299. Zash

    Prosody doesn't have that, so it doesn't exist!

  300. arc

    Given that everything in this community takes 10 years or more to come about

  301. jonas’

    arc: who is "we"?

  302. arc

    We in this room right now, I would hope

  303. arc

    And this community as in XML community. Given that we are still stuck with XPath/xslt/etc in browsers now in 2021

  304. Zash

    I'd refer back to what Kev said

  305. arc

    XSLT is perfectly fine stop hating on it 😋

  306. stpeter has joined

  307. stpeter has left

  308. Andrzej

    arc, would exi work for transforming XML stanza into "payload" for the client?

  309. Andrzej

    maybe I should be thinking about EXI instead of XSLT

  310. Steve Kille has left

  311. Kev has left

  312. Zash

    I am, of course, referring to https://logs.xmpp.org/xsf/2021-02-11?p=h#2021-02-11-521589a72d0ff42a

  313. Kev has joined

  314. Kev has left

  315. Kev has joined

  316. matkor has left

  317. matkor has joined

  318. arc

    No. EXI is just an alternative representation of XML data. It does make XDM/XPath/Xquery/XSLT faster when written with it in mind. But all processing is generally faster with the EXI because string compare is just that much slower

  319. Ge0rG

    I'm sure there is nothing wrong with executing attacker-provided XSLT on your XMPP server.

  320. arc

    I never said processing XSLT provided by the user.

  321. arc

    XSLT is a turing complete language. You can open a similar glaring security hole by running user provided python, or most languages for that matter.

  322. Zash

    (XEP-0060 has references to XSLT 😱️)

  323. Andrzej

    also (as it was mentioned that I would like to "encoding the limits of google and apple services into my server"), I think that transformation and encoding is OK and would not impose limits on your server and if client would pass "limit" of the payload which it can receive, then server would just respect request from the client

  324. stp has left

  325. stp has joined

  326. Zash

    So, size constraint. How about an ordered-by-priority list of payloads you're interested in? Server strips anything not in that list, then if it's still too large, strips the lowest priority payloads until it's small enough?

  327. Zash

    Where "list of payloads" could be xpath or just namespaces or somesuch, details.

  328. Andrzej

    that "could" work

  329. Adi has left

  330. krauq has left

  331. arc

    Andrzej: this is a simplistic but valid way of thinking about EXI; within the root of a XMPP stream you typically only find about 6-10 elements; obviously your IQ, MESSAGE, and PRESENCE elements (capitalized only for clarity), stream: namespace stuff, and possibly session management.

  332. krauq has joined

  333. arc

    Text domain XML would have the xml parser run multiple passes over the data stream to find delimiters (space, brackets, quotes, etc), break the stream up into events, possibly create a minidom out of those events, and then pass qnames (namespaces, prefixes, element names, attribute names, etc) to the application typically as string pointers into the stream buffer. But increasingly for memory safety, memcpy all those strings into new buffers too.

  334. arc

    And then no matter how the client or server handles stanza routing, it either ends up testing the qnames against a large list of qnames it's designed to pass to various functions, or it hashes the qnames and checks for a value on a hashmap. In every case there's thousands of machine instructions for every stanza.

  335. arc

    Going back to those, lets say always 15 or less possible elements found in a <stream:stream> root, EXI represents that as a binary number which typically is either bitpacked with only four bits used, or structured for later compression. In every case, you don't get "message", you just get the binary number 1. Which is much much simpler to parse.

  336. arc

    Ignoring the stuff you can read up on if you ever wanted to implement it, that is EXI in a nutshell. And that is why it is not just 25x or more faster than text XML processing, it also takes far less memory and can be much more easily squeezed onto a microcontroller.

  337. Andrzej

    ok, I get it now, thanks

  338. Kev

    Has anyone tried doing EXI for S2S?

  339. arc


  340. x51 has joined

  341. Andrzej has left

  342. Kev

    Hmm. Going from https://www.w3.org/XML/EXI/#Efficient_XML I find only one (commercial) EXI implementation for C/C++ that doesn’t claim to be alpha quality.

  343. Zash

    Someone did mention a lack of libraries earlier.

  344. Kev


  345. andrey.g has left

  346. arc

    Transforming stanzas between exi grammars is not difficult provided your grammar mapping is set up correctly

  347. Kev

    arc: Was that responding to me? Because if so I don’t see how it relates :)

  348. Kev

    arc: Was that responding to my lack of library comment? Because if so I don’t see how it relates :)

  349. stpeter has joined

  350. stpeter has left

  351. pasdesushi has joined

  352. arc

    Using it for s2s, you're typically going to be routing stanzas between grammars. So in the above example, a xmpp client might say a message=1, but you may route that stanza to a server using a grammar that says message=0

  353. Kev

    Ah, my previous question, gottit, thanks.

  354. pasdesushi has left

  355. pasdesushi has joined

  356. arc

    With text domain xmpp you often don't have to modify the stanza. And when you do, only in small specific ways like changing the from= attribute. But with EXI S2S is also grammar conversion.

  357. arc

    But going back to the xpath discussion, XDM 3.0 was certainly designed with EXI in mind though it is not referenced directly. EXI does not solve any problems in this area, it only makes things faster and with less bandwidth.

  358. Kev

    After a few minutes of looking, I’ve come to the conclusion that there’s no implementation for C/C++ that doesn’t claim to be alpha/seem abandoned other than one commercial Windows-only one. And, as you asserted earlier, I don’t want to be working on an EXI parser :)

  359. krauq has left

  360. arc

    There are more, and yes they are commercial. Thankfully no one has released a free one that is ready to use or I would start having trouble paying my rent.

  361. winfried has joined

  362. pasdesushi has left

  363. krauq has joined

  364. arc

    Anyways all that is simply different problem. It doesn't help with identifying stanzas for routing to external software, whether you want to call that microservices or not.

  365. Ge0rG

    Re EXI I think we could have significant benefits on mobile / low bandwidth, if a client implementation creates a grammar of everything supported by the client, uploads it to the server in some secure way, and the server only ever uses elements within that grammar to the client, stripping everything unknown and dropping empty elements.

  366. Ge0rG

    Your client doesn't support CSNs? It's omitted from the grammar, server strips out the element from messages, empty messages get dropped on the server, battery wins

  367. pasdesushi has joined

  368. pasdesushi has left

  369. pasdesushi has joined

  370. xecks has left

  371. jonas’

    except only proprietary implementations, so no chance there

  372. xecks has joined

  373. emus has left

  374. moparisthebest has left

  375. pasdesushi has left

  376. pasdesushi has joined

  377. derdaniel has left

  378. derdaniel has joined

  379. pasdesushi has left

  380. pasdesushi has joined

  381. Andrzej has joined

  382. Mikaela has joined

  383. pasdesushi has left

  384. pasdesushi has joined

  385. govanify has left

  386. govanify has joined

  387. pasdesushi has left

  388. pasdesushi has joined

  389. alameyo has joined

  390. Ge0rG

    jonas’: implementations of EXI?

  391. Ge0rG

    How hard can it be?

  392. jonas’

    how hard can a binary xml parser be?

  393. stpeter has joined

  394. stpeter has left

  395. Steve Kille has joined

  396. Steve Kille has left

  397. Steve Kille has joined

  398. Kev

    I mean, you just take an XML parser and you change some strings to numbers, right? Job done ;)

  399. Zash

    That sounds fun, we could call it FunXMPP

  400. Kev


  401. Kev


  402. intosi has left

  403. derdaniel has left

  404. derdaniel has joined

  405. Adi has joined

  406. raghavgururajan has left

  407. floretta has joined

  408. LNJ has left

  409. raghavgururajan has joined

  410. Ge0rG

    it's not-fun, so let's call it NunXMPP. Will also make clear that it's neither fun nor sexxy.

  411. alameyo has left

  412. moparisthebest has joined

  413. LNJ has joined

  414. pasdesushi has left

  415. intosi has joined

  416. floretta has left

  417. antranigv has left

  418. stpeter has joined

  419. stpeter has left

  420. derdaniel has left

  421. derdaniel has joined

  422. pasdesushi has joined

  423. derdaniel has left

  424. derdaniel has joined

  425. antranigv has joined

  426. derdaniel has left

  427. derdaniel has joined

  428. edhelas has left

  429. edhelas has joined

  430. intosi has left

  431. derdaniel has left

  432. derdaniel has joined

  433. pasdesushi has left

  434. pasdesushi has joined

  435. pasdesushi has left

  436. pasdesushi has joined

  437. intosi has joined

  438. derdaniel has left

  439. derdaniel has joined

  440. floretta has joined

  441. Steve Kille has left

  442. Steve Kille has joined

  443. wladmis has left

  444. nyco has left

  445. wladmis has joined

  446. pasdesushi has left

  447. pasdesushi has joined

  448. wladmis has left

  449. wladmis has joined

  450. Steve Kille


  451. Steve Kille


  452. Steve Kille

    ignore me

  453. Zash

    "push 'f' to pay respects"?

  454. wladmis has left

  455. wladmis has joined

  456. wladmis has left

  457. wladmis has joined

  458. stpeter has joined

  459. stpeter has left

  460. papatutuwawa has joined

  461. floretta has left

  462. krauq has left

  463. nyco has joined

  464. derdaniel has left

  465. derdaniel has joined

  466. krauq has joined

  467. papatutuwawa has left

  468. wladmis has left

  469. papatutuwawa has joined

  470. floretta has joined

  471. derdaniel has left

  472. derdaniel has joined

  473. alameyo has joined

  474. Wojtek has left

  475. Andrzej has left

  476. pasdesushi has left

  477. pasdesushi has joined

  478. papatutuwawa has left

  479. stpeter has joined

  480. stpeter has left

  481. Andrzej has joined

  482. wladmis has joined

  483. Wojtek has joined

  484. lskdjf has joined

  485. wladmis has left

  486. wladmis has joined

  487. emus has joined

  488. wladmis has left

  489. derdaniel has left

  490. derdaniel has joined

  491. stpeter has joined

  492. stpeter has left

  493. pasdesushi has left

  494. pasdesushi has joined

  495. wladmis has joined

  496. derdaniel has left

  497. derdaniel has joined

  498. pasdesushi has left

  499. pasdesushi has joined

  500. Daniel has left

  501. pasdesushi has left

  502. pasdesushi has joined

  503. pasdesushi has left

  504. derdaniel has left

  505. pasdesushi has joined

  506. derdaniel has joined

  507. adiaholic has left

  508. adiaholic has joined

  509. chronosx88 has left

  510. chronosx88 has joined

  511. chronosx88 has left

  512. chronosx88 has joined

  513. serge90 has left

  514. wladmis has left

  515. LNJ has left

  516. serge90 has joined

  517. andy has left

  518. LNJ has joined

  519. winfried has left

  520. wladmis has joined

  521. Adi has left

  522. stpeter has joined

  523. stpeter has left

  524. wladmis has left

  525. pasdesushi has left

  526. pasdesushi has joined

  527. Adi has joined

  528. Daniel has joined

  529. wladmis has joined

  530. nyco has left

  531. wladmis has left

  532. wladmis has joined

  533. nyco has joined

  534. LNJ has left

  535. LNJ has joined

  536. LNJ has left

  537. LNJ has joined

  538. arc has left

  539. arc has joined

  540. matkor has left

  541. matkor has joined

  542. arc has left

  543. arc has joined

  544. arc has left

  545. arc has joined

  546. alameyo has left

  547. alameyo has joined

  548. Daniel has left

  549. nyco has left

  550. pasdesushi has left

  551. pasdesushi has joined

  552. matkor has left

  553. pasdesushi has left

  554. pasdesushi has joined

  555. matkor has joined

  556. alameyo has left

  557. pasdesushi has left

  558. pasdesushi has joined

  559. nyco has joined

  560. marek has left

  561. pasdesushi has left

  562. pasdesushi has joined

  563. derdaniel has left

  564. derdaniel has joined

  565. marek has joined

  566. derdaniel has left

  567. derdaniel has joined

  568. arc

    You do not need to even parse. All you really need is some basic data structures. EXI is extremely easy to implement a reader and writer for.

  569. neshtaxmpp has left

  570. arc

    The difficult part is generating grammar based on schema, then generating code based on grammar. And you'll see that for most cases, this is the part that is not provided

  571. debacle has left

  572. arc

    I have seen some foss implementations that work fine, but require that you provide a grammar file. Or provide all the low level data structures and the header, but leave the application api to build your own EXI decoder/encoder.

  573. moparisthebest

    what are the upsides? just that it's ever-so-slightly smaller?

  574. jonas’

    moparisthebest, savings by EXI are more than just "ever so slightly". especially when base64’d stuff comes into play. my understanding is that you could transfer that in decoded form in EXI

  575. moparisthebest

    which saves what, 33% only in the case of base64'd stuff?

  576. jonas’

    that’s considerable on GPRS

  577. jonas’

    and remember that all of OMEMO is base64’d

  578. moparisthebest

    GPRS is dead though so who cares

  579. jonas’

    you wish

  580. derdaniel has left

  581. moparisthebest

    actually I wish it wasn't, but that's beside the point :)

  582. moparisthebest

    in the USA 2G has been gone awhile, most carriers dropped 3G last month, the rest will next year

  583. neshtaxmpp has joined

  584. moparisthebest

    is all EXI offers "better compression" then ? and if so, how does it compare to "just compressing" XML, and does it suffer from the same security vulnerabilities "just compressing" XML does ?

  585. SamWhited

    moparisthebest: that doesn't mean 2G is dead, that means the major carriers dropped it and if you were in an area where it no longer exists you just don't have internet anymore unless you switch to a smaller local provider or use a slow wireless uplink. That's *more* of a reason to save bandwidth, not less.

  586. chronosx88 has left

  587. chronosx88 has joined

  588. SamWhited

    I make no comment on EXI in particular, just complaining because I'm always annoyed when I've lived somewhere where it's impossible to get fast internet and people say "what's the problem, everywhere has fast broadband now!"

  589. moparisthebest

    I think we might have discussed this before, and I think the conclusion was EXI is probably as vulnerable to CRIME/BREACH as gzip is ?

  590. moparisthebest

    SamWhited, yea I think it sucks, but it doesn't change the fact that in the USA (and soon everywhere else presumably), it's dead

  591. moparisthebest

    I think next year for canada iirc

  592. SamWhited

    I don't thik that's true though, it's just dead if you use AT&T or T-Mobile, otherwise I believe smaller providers are still using it because it's what they can afford to do easily. Even if it is dead, that doesn't mean "we don't need to worry about a 33% bandwidth savings" because it being dead doesn't mean everyone gets upgraded to 4G

  593. alameyo has joined

  594. nyco has left

  595. LNJ has left

  596. moparisthebest

    right, I'm not saying EXI is good or bad, I'm asking what advantages it offers, how it compares to just running normal compression on XML etc

  597. neshtaxmpp has left

  598. SamWhited

    Maybe I misunderstood "which saves what, 33% only in the case of base64'd stuff? / GPRS is dead though so who cares"

  599. Zash

    I vaguely recall all of Sweden having some kind of GSM coverage at some point, but since recent-G networks, not so much. Not much profits in covering the Scandinavian mountain range in high-speed internet.

  600. SamWhited

    That sounds like "we don't have to save bandwidth, everyone has fast internet now"

  601. Zash

    AIUI you get rid of the parser and shuffle packed structs over the wire, instead of a text format that requires a proper parser.

  602. marek has left

  603. moparisthebest

    that sounds like a proper security nightmare to me

  604. marek has joined

  605. Zash

    How's HTTP/2 doing? How's all the ... binary Google format I don't remember the name of anymore.

  606. moparisthebest

    so for sure EXI has downsides compared to compressed XML, like comparatively few libraries, no/much less security auditing, and such, does it have upsides?

  607. Daniel has joined

  608. andy has joined

  609. Zash

    Compression of the zlib variety has memory usage issues and security .. ickyness.

  610. moparisthebest

    Zash, HPACK ? https://tools.ietf.org/html/rfc7541 https://blog.cloudflare.com/hpack-the-silent-killer-feature-of-http-2/

  611. Zash

    EXI supposedly doesn't have that

  612. moparisthebest

    how sure is anyone EXI doesn't have the same security problems ?

  613. Zash

    I'm sure it's just as good as ASN.1 & co! 😛

  614. LNJ has joined

  615. Zash

    moparisthebest, protocol buffers, was the thing I forgot the name of. I mentally included HPACK in "HTTP/2"

  616. SamWhited

    I don't remember much about EXI, but it negotiates the cmopression up front instead of constructing a dictionary that is reused across stanzas right? In that case I don't see how it could be possible for it to have the same issues as stream compression or TLS compression

  617. moparisthebest

    even HPACK acknowledges it's vulnerable to CRIME and friends https://tools.ietf.org/html/rfc7541#section-7.1.1

  618. Zash

    Can't you ... not do that with EXI?

  619. Zash

    Just do the binary packing stuff

  620. Zash

    CRIME & co comes from compressing user/attacker data in the same context as other stuff, so if you just don't do that, things should be better

  621. moparisthebest

    SamWhited, I realize current-xmpp-compression doesn't do this, but nothing stops you from designing a XEP such that each stanza is compressed separately right?

  622. stpeter has joined

  623. stpeter has left

  624. Zash

    moparisthebest: Correct, just need to write it down somewhere and get implementations to comply.

  625. SamWhited

    moparisthebest: sure, I've worked plces that implemented it that way in the pas

  626. SamWhited


  627. moparisthebest

    so, that's *not* vulnerable to CRIME then ?

  628. Steve Kille has left

  629. moparisthebest

    then the questions are: 1. is EXI vulnerable to CRIME ? 2. if not, how does it compare size-savings-wise to just doing the above with XML ?

  630. jonas’

    I did some measurements on the impact of compressing stanzas individually some time ago, and it was "still worth it"

  631. SamWhited

    Right. Assuming you're not doing something weird with your stanzas where you're mixing secret data and non-secret data, but in the general sense that's right

  632. jonas’

    I’m not sure if EXI contains any compression at all

  633. Zash

    moparisthebest, doing a full flush between each stanza? yes, that fixes CRIME (if I remember correctly what CRIME was about)

  634. jonas’

    or if you have to use it

  635. moparisthebest

    in theory it should get rid of basically all base64 overhead right?

  636. neshtaxmpp has joined

  637. Zash

    IIRC you only need to do a full flush when the sender changes.

  638. Steve Kille has joined

  639. alameyo has left

  640. LNJ has left

  641. Zash

    Might be interesting things you could do with CSI integration

  642. neshtaxmpp has left

  643. SamWhited

    We just did a full flush on stanza boundaries at HipChat. Probably could have saved more by being clever, but it was easy and it helped a lot. Dropped our network traffic by a factor of 0.58 and dropped CPU utilization by a factor of 0.60

  644. LNJ has joined

  645. SamWhited

    (with ZLIB)

  646. Zash

    A big part of why we dropped compression completely from Prosody was memory usage tho.

  647. moparisthebest

    might even be better options today, it'd be interesting to see XMPP-wise how zstd and brotli compared

  648. nyco has joined

  649. SamWhited

    Yah, I didn't write down or publish our memory usage, but I assume it went up a bit. Don't remember it though, had to go look up those two numbers. It's been a while.

  650. chronosx88 has left

  651. chronosx88 has joined

  652. Zash

    I'd be interested in fixed-dictionary compression

  653. moparisthebest

    iirc that's what EXI is doing ^ ?

  654. moparisthebest

    except it negotiates the dictionary, roughly

  655. Kev

    That’s more or less what EXI is isn’t it?

  656. Kev

    Heh, beaten :)

  657. Zash

    I though it was more like bit packing structs, not like zlib & co

  658. moparisthebest

    but that implies doing the same compression across all stanzas in a stream, which probably implies CRIME

  659. Zash

    Not if you don't allow backreferences into user data, only into the dictionary

  660. moparisthebest

    or possibly some other attack if you know the dictionary used, idk

  661. moparisthebest

    isn't "xml element names used in the stanza" sometimes "user data" ?

  662. Zash

    You don't learn anything about the previous stanza sent by someone else

  663. Zash

    If you say "hello", then an attacker saying "hello" right after would be smaller because it can reference that previous "hello"

  664. Zash

    But if you build a static dictionary, that everyone involved already know, you don't leak user data from that.

  665. Zash

    Static dictionary would be something full of angle brackets and protocol stuff, no user data.

  666. moparisthebest

    I don't know, maybe? https://tools.ietf.org/html/rfc7541#section-7.1.2

  667. nyco has left

  668. SamWhited

    I've thoguht about doing that with zstd a couple of times. It has a training mode where you can give it a sample set and it builds a dictionary from that you can reuse later. Never actually tried to see how well it does with a big set of XML though.

  669. nyco has joined

  670. pasdesushi has left

  671. pasdesushi has joined

  672. winfried has joined

  673. Andrzej has left

  674. serge90 has left

  675. benharri has left

  676. benharri has joined

  677. serge90 has joined

  678. paul has left

  679. Wojtek has left

  680. derdaniel has joined

  681. paul has joined

  682. derdaniel has left

  683. derdaniel has joined

  684. antranigv has left

  685. derdaniel has left

  686. derdaniel has joined

  687. Adi has left

  688. Adi has joined

  689. Andrzej has joined

  690. antranigv has joined

  691. wladmis has left

  692. arc has left

  693. arc has joined

  694. arc

    moparisthebest: is more than a little compression. With xmpp it can transform the overhead of a <message> stanza from around 100-200 bytes (depending on jid length) to around 10.

  695. moparisthebest

    arc, so is it vulnerable to CRIME, and how does that compare to simply compressing XML stanzas

  696. arc

    Where did you get that idea?

  697. moparisthebest

    that was a question, not a statement :) "is it"

  698. arc

    I don't believe so, no. Because it is not compression. It would use the same number of bits regardless of the length of the qnames, jids, etc.

  699. derdaniel has left

  700. derdaniel has joined

  701. arc

    EXI is a binary representation of XML. One of the functions is designed for is to then run it through a conventional compression such as DEFLATE

  702. derdaniel has left

  703. derdaniel has joined

  704. derdaniel has left

  705. derdaniel has joined

  706. arc

    Another option is to bitpack.

  707. derdaniel has left

  708. derdaniel has joined

  709. moparisthebest

    is *that* vulnerable to CRIME

  710. arc

    https://www.w3.org/TR/exi-primer/ goes into the higher level nitty gritty if you want to read more.

  711. arc

    No because it is constant width.

  712. derdaniel has left

  713. derdaniel has joined

  714. derdaniel has left

  715. derdaniel has joined

  716. pasdesushi has left

  717. pasdesushi has joined

  718. derdaniel has left

  719. derdaniel has joined

  720. arc

    The most you could probably learn from CRIME is encoding type. EXI supports four modes; byte packed, bit packed, precompressed, or compressed.

  721. derdaniel has left

  722. derdaniel has joined

  723. arc

    And the letter two are basically the same except the compression is deflate added on top of precompressed

  724. arc

    https://www.w3.org/TR/exi-primer/#compression explains how data in the stream is rearranged for better compression

  725. moparisthebest

    so how does it do on a super simple message stanza like https://paste.rs/Oe6.xml ?

  726. moparisthebest

    $ wc -c * 175 msg.xml 101 msg.xml.brot 146 msg.xml.gz 130 msg.xml.zst

  727. arc

    That will depend a lot on the mode

  728. Kev

    And the schema, presumably.

  729. moparisthebest

    brotli does surprisingly well here (no settings touched, just `brotli -c msg.xml > msg.xml.brot`)

  730. arc

    But you can basically ignore everything inside the XML brackets

  731. moparisthebest

    not the attribute values?

  732. arc

    Everything inside the XML brackets

  733. moparisthebest

    so, run some common examples maybe?

  734. arc

    There is an example in the primer

  735. moparisthebest

    of what, say a normal chat client might use for a mode/grammar/whatever

  736. Daniel has left

  737. arc

    https://www.w3.org/TR/exi/#stringTable will take care of the jids

  738. arc

    <message becomes one byte (except in bit packed), SE-MESSAGE

  739. moparisthebest

    > The life cycle of a string table spans the processing of a single EXI stream.

  740. moparisthebest

    so that likely makes it vulnerable to CRIME then ?

  741. moparisthebest

    in fact, it certainly would right?

  742. moparisthebest

    if you put JIDs in there anyhow

  743. arc

    from= and to= are each AT, so one byte each since they are optional. The values come from the string value table

  744. marek has left

  745. moparisthebest

    except they aren't because you can't do that and avoid CRIME

  746. derdaniel has left

  747. derdaniel has joined

  748. moparisthebest

    so add them back in

  749. marek has joined

  750. arc

    What is your obsession with CRIME? The string tables are a constant value. They are not susceptible

  751. moparisthebest

    because you can't just go "maybe it's not vulnerable to breaking the stream encryption ¯\_(ツ)_/¯"

  752. derdaniel has left

  753. derdaniel has joined

  754. moparisthebest

    it either is, or it is not

  755. arc

    The only thing you're going to get for information leak from the string table is the number of values in the table.

  756. moparisthebest

    attackers can manipulate the JIDs that go across the stream right ?

  757. Tobias has left

  758. Tobias has joined

  759. arc


  760. moparisthebest

    how would any JID except the sending one be in the dictionary up front anyhow ?

  761. Daniel has joined

  762. moparisthebest

    how will you know randomjid@randomdomain is going to message you?

  763. moparisthebest

    maybe EXI is only useful in closed deployments where you know the entire network up front ?

  764. moparisthebest

    (and, maybe, don't care about TLS providing CRIME-proof security, still unknown on this one)

  765. arc

    You're right the length of jid could be leaked.

  766. moparisthebest

    so if you cut that stanza down to essentially the bare minimum info that must be transmitted in it, and it's probably *too* cut down actually, you end up with something like https://paste.rs/aPt which is 86 bytes

  767. mathijs has left

  768. moparisthebest

    that means EXI would have to fall somewhere between 86 and brotli's 101 to be useful at all

  769. moparisthebest

    that's a small range for improvement

  770. Daniel

    i think that's a very simplistic example

  771. Daniel

    look at an omemo pre key bundle for example

  772. Daniel

    or generally anything with lots of nasted elements

  773. moparisthebest

    sure, I suspect brotli would improve even more with anything base64'd though

  774. Daniel

    plus 184 requests 333 requests

  775. Daniel

    and all the other stuff we regulary put into messages

  776. Daniel

    and it should be easier on the cpu

  777. moparisthebest

    honestly regular compression's ratio should get better the bigger the stanza is right? so a small one like this is probably least fair

  778. Daniel

    i have a deployment where we would like to use compression but can’t because it's too expensive

  779. pasdesushi has left

  780. pasdesushi has joined

  781. Daniel

    and if implemented correctly exi can even be faster than string parsing

  782. Daniel

    where as compression is always slower

  783. arc

    You can add brotli on top of EXI. It works very well for that purpose.

  784. moparisthebest

    I doubt that's always true (that compression is slower)

  785. SamWhited

    Daniel: are you sure about that? We had CPU usage drop on one machine because there were less TLS packets where most of the CPU was being taken up

  786. arc

    But again this is all based on the grammar.

  787. Daniel

    SamWhited, no

  788. SamWhited

    Worth measuring anyways if you haven't. If you're using a slow TLS cipher and a fast compression algorithm you might make some gains

  789. moparisthebest

    I understand EXI should be able to be better than general compression in a closed system where you know every stanza that will ever be passed

  790. moparisthebest

    the question is, can EXI be better than general compression for a chat client in the public federated XMPP network

  791. larma

    any compression system that works on reusing user input is has issues the like of CRIME and similar. EXI could be fine if no string lookup table was used. However schema-based EXI is complicated and schema-less EXI is far from optimal

  792. larma

    any compression system that works by reusing user input has issues the like of CRIME and similar. EXI could be fine if no string lookup table was used. However schema-based EXI is complicated and schema-less EXI is far from optimal

  793. ti_gj06 has left

  794. SamWhited

    I suspect the question is actually "assuming that EXI and GZIP flushing on stanza boundaries are both 'good enough', which one is easier to implement and deploy widely" (and the answer is probably normal stream compression), but of course I may be wrong.

  795. moparisthebest

    yes roughly SamWhited , I think it's rather obvious general compression and regular XML is easier no ?

  796. larma

    somthing like exi would still be more efficient than gzip. also why would you think flushing on stanza boundaries is enough?

  797. moparisthebest

    so it becomes 1. is EXI secure 2. is it better than that, and if so, better enough to justify the effort?

  798. Zash

    Could do something in the direction of HPACK and/or WhatsApps FunXMPP compression scheme...

  799. SamWhited

    moparisthebest: I suspect so, yes

  800. arc

    moparisthebest: again, EXI is not compression. It is a schema-aware binary representation of XML data. It is intended that you use compression on top of it.

  801. arc

    When you're not using compression on top of it, typically you use bit packed mode.

  802. adiaholic has left

  803. adiaholic has joined

  804. moparisthebest

    then you'd have to show EXI+compression is better in terms of size+effort vs XML+compression ?

  805. moparisthebest

    and additionally, ensure EXI doesn't introduce security bugs like CRIME

  806. flow

    larma> somthing like exi would still be more efficient than gzip. also why would you think flushing on stanza boundaries is enough? honest question: why would you think that it is not enough?

  807. arc

    Actually as I'm thinking about it, it would make a lot of sense to pre-populate the values string table with the JIDs in your roster. And if you are worried about CRIME, you can also specify that padding is used to make all jids transfer as a fixed width

  808. larma

    flow, attackers can control certain content of stanzas even if it's not them sending

  809. moparisthebest

    so then you can use than as an oracle to determine a user's roster

  810. arc


  811. flow

    larma, unfortunately I am appearantly missing pieces why this is relevant

  812. larma

    ah maybe there is something that I have in mind that nobody else has. If we ever do some kind of efficient compression on stanzas, I'd want to also make use of it inside SCE. Then it's no longer transport encryption only, but also about end-to-end-encryption relevant. And if you assume the server as an attacker, they can easily modify IDs and similar to modify certain parts of your message to exfiltrate others.

  813. larma

    ah maybe there is something that I have in mind that nobody else has. If we ever do some kind of efficient compression on stanzas, I'd want to also make use of it inside SCE. Then it's no longer transport encryption only, but also about end-to-end-encryption relevant. And if you assume the server as an attacker, they can easily modify IDs and similar to modify certain parts of your encrypted message to exfiltrate others.

  814. flow

    larma, thanks, I probably need to think about this a little more (and with more sleep)

  815. SamWhited

    I still think that doesn't seem worth considering, the server has to be trusted, that's the whole model that we have. Trying to change that and doing complicated partial stanza encryption just seems like a waste of time. Let's just settle on compression that's "good enough" and in the occasional system where there's some high-security environment where the server isn't trusted, don't use it.

  816. Zash

    Did anyone define "good enough"?

  817. flow

    larma, but it sounds like you are talking about an intermediate hop modifyin encrypted bytes, that shouldn't be possible, no?

  818. SamWhited

    Zash: "better than no compression?"

  819. arc

    I have to agree with Sam

  820. arc

    Here's the thing; XMPP is low enough bandwidth and no one really cares except IoT, and maybe mobile.

  821. emus

    (Different topic: Kev - there was a misunderstanding with Sam W., nevermind about the Twitter thing - I will just request the pinning via mail 😊️ )

  822. larma

    flow, no, certain parts are unencrypted, like message id. Also you very likely would want to be able to reply to unencrypted messages with encrypted messages without risking to actually leak the content of the encrypted message 😉

  823. SamWhited

    Yah, sorry, I didn't realize Twitter required me to have a separate personal account

  824. SamWhited

    I'm still happy to help on the comms team as necessary though, someone just tell me what needs doing and I'll see what I can do.

  825. flow

    larma, ahh I think I got it now: values that are to-be encrypted are determined by the server

  826. arc

    On IoT these days we mostly care about making xml easy enough to use that we can fit it on a microcontroller and utilize less cpu and transmit size so that the battery lasts longer

  827. flow

    that is certainly interesting. I never considered compressing the encrypted bytes prior base64 encoding them and the implications of doing so

  828. emus

    SamWhited if you read in the CommTeam MUC occasional or subscribe to the PR that should be fine. If you want, I can point you as reviewer there, too

  829. arc

    EXI seems extremely complicated, but CPUs can process it with a minimum amount of code. A typical EXI compiled binary static library is around 8k.

  830. flow

    FWIW, I believe EXI would also be able to avoid base64 for raw bytes (at least, that is what I remember being told ~5 years ago)

  831. SamWhited

    emus: can you invite me? I don't see the comms team muc listed on the website

  832. emus

    https://xmpp.org/newsletter.html here is a link if thats okay for you, I dont have you in my contact list so I guess thats quicker

  833. SamWhited


  834. Andrzej has left

  835. arc

    And when you're dealing with a microcontroller costing under $1, that only has like 16k-64k flash for all the software, the size of the software matters a lot. And when the manufacturer is budgeting the mAh battery, they care a lot about keeping the microcontroller in sleep and moreso keeping the transmitter powered down as much as possible when the cpu is awake.

  836. inky has joined

  837. arc

    Because in the end, they don't care about anything else. They don't care about standards compliance. They don't care what technical solution is used. They don't care about CRIME or data security.

  838. arc

    They're making 10 million of these units, at minimum, so if they can save 10 cents that's a million dollar profit.

  839. arc

    They look at a big stack of money on one side, and a technician whining about standards compliance or user security on the other, they will choose the big stack of money every time.

  840. arc

    The only reason *some* of these manufacturers are using xml at all, and a very small number overall, is because exi can do the work with less code, less cpu, and less transmit power than message queues, which is what they would be using otherwise

  841. neshtaxmpp has joined

  842. moparisthebest

    right, I get why EXI is a win for bottom-of-the-barrel iot, still not sure if it's a win or not for generic federated xmpp chat client though

  843. arc

    Nobody is using it for general federated xmpp

  844. moparisthebest

    yes but I'm wondering if it's worth pursing it for that though

  845. Zash

    moparisthebest, ~1MB chunks of XML sent to my phone something something would be nice if it was smaller

  846. moparisthebest

    but maybe regular compression solves it for you, and that's easy

  847. arc

    Half the time they just want http anyway. Because the software is free and HTTP is lightweight. And because their in-house developers already know it.

  848. Zash

    but then most of that consists of people sending presence to all the same MUCs that I'm in

  849. arc

    Zash, I agree with you. And it would likely turn the 1MB chunk into 1kb.

  850. arc

    String tables are surprisingly efficient at jids

  851. Zash

    FWIW this is with CSI buffering up unimportant stuff into larger chunks, that's how it gets so large.

  852. moparisthebest

    arc, but "string tables" can't be used on public federated XMPP because they are vulnerable to CRIME, so they are out

  853. fuana has joined

  854. arc

    I would say that you choose not to use it because you believe is impossible to implement with your security criteria in mind

  855. Zash

    the real question is, is CRIME really that bad?

  856. arc

    And if I really cared I would have worked on it a long time ago

  857. SamWhited

    yah, honestly, after SASL is complete I'm not sure how much I care about CRIME-like vulnerabilities.

  858. Zash

    IIRC you almost need to launch a DoS attack to get anything out of it, maybe just sprinkling some rate limits on top makes it all Good Enough

  859. arc

    I am certain that there is an orchard full of hanging fruit for xmpp security, nobody will go through the effort

  860. jonas’

    CRIME also gets less and less relevant with e2ee

  861. jonas’

    (post-SASL that is)

  862. Zash

    And good luck doing CRIME with SCRAM anyways

  863. arc

    I mean, all the software that uses libxml2 should just be considered insecure by that very nature

  864. moparisthebest

    makes me pretty nervous to pretend CRIME isn't a threat... attacks only get better, never worse

  865. arc

    Google was so certain of the fundamental insecurity of anything XML that they never even implemented encrypted S2S

  866. Zash

    Huh. Source?

  867. moparisthebest

    and actually I keep saying CRIME when BREACH is actually a better fit for what we are talking about https://en.wikipedia.org/wiki/BREACH

  868. arc

    Zash, source is the number of times I've had beer with Google developers

  869. Zash

    moparisthebest, C-ompression something something is easy to remember tho

  870. fuana has left

  871. fuana has joined

  872. arc

    In 2019 I had a drink with a few of the guys that previously worked on the gtalk/hangouts sre team, and I commented that it probably was a lot easier doing seculity with Go and it's brand new xml library. One of them shot his drink out his nose

  873. arc

    They apparently don't use XML internally anymore because of it

  874. moparisthebest

    well, google is well known for their poor decisions so :)

  875. moparisthebest

    imagine taking advice from a company that created and killed 83 chat systems in the last 15 years

  876. arc

    But that's the thing, not once did they create one that actually works the way they wanted it to

  877. arc

    And anything even remotely stable is just piled on mess over mess until they have to start over

  878. SamWhited

    They are spot on with that one though, XML is absolute garbage in terms of being able to do a secure implementation. Way to big of an attack surface for something that should just be a way to transfer a tree…

  879. SamWhited

    Not that Google's dev practices or product management practices are always great, but they know their security.

  880. StefanK$ has joined

  881. arc

    Protobuffs is wide open to attack too. They work around that with things like ucs4 and fixed length string entries

  882. SamWhited

    I'm not saying protobuf is perfect, I don't know much about them, just that they're not wrong about XML being bad.

  883. moparisthebest

    XML is the worst except for everything else

  884. SamWhited

    No, it's just worse than most other things. We're stuck with it for the base of XMPP, and I don't see anything else that works in a similar enough way to do a streaming protocol in as nice of a manner, but that doesn't mean it's not insecure garbage or that we shouldn't be *very* careful with it

  885. arc

    XML is not bad by design. Is bad because nobody cares.

  886. DebXWoody has joined

  887. fuana has left

  888. Yagiza has left

  889. SamWhited

    I don't think that's true. Designs like hacking namespaces on and mixing them with attributes, adding things like proc insts, etc. are just bad.

  890. arc

    I'm not saying that's not bad. I'm saying the XML suffers the same type of abandonment that led to heartbleed

  891. papatutuwawa has joined

  892. SamWhited

    I mean, I'm sure nobody caring is a problem too, but I'd also say XML is just bad by design and therefore we need to be *very* careful about its use and not dismiss other peoples concerns about it so redily.

  893. Andrzej has joined

  894. SamWhited

    I just get mad every time somebody does the "it's just <> vs {}, why is HN so trendy?" or whatever in this room when realistically whatever they're complaining about is probably a serious problem that we should be addressing instead of being dismissive.

  895. moparisthebest

    but the "json is better than xml" people don't have any answers for all the ways in which json is worse

  896. moparisthebest

    and you can s/json/anything/ there too

  897. Zash

    My langsec friend said something some time that I remember as "xml is okay, it's not made of length-prefixes and stuff"

  898. SamWhited

    Sure, but that's not the problem, the problem is that we pretend that means XML is good somehow and then Go has the issue where namespaces can be manipulated and it's like the third time I've seen that in an XML decoder and somehow we just say "no, XML is fine, let's use more of it" every damn time

  899. Kev

    The fundamental problem is that XML got so much stuff *right* that we’re stuck with the stuff it didn’t.

  900. moparisthebest

    "bad libraries exist" is a thing

  901. SamWhited

    "Every library is consistently bad in the same ways" is the actual problem.

  902. arc

    Oh I'm not dismissing anything. I'm just worn down by an industry that doesn't care until it nearly destroys them. I lead the Python xml-sig that produces code responsible for billion dollar industries. Those companies don't care. If the problem were made bluntly clear to them, they wouldn't fund anything. They would direct their technical teams to migrate to another language

  903. moparisthebest

    binary formats never have parsing vulnerabilities https://duckduckgo.com/?q=asn.1+parsing+vulnerabilities

  904. SamWhited

    But I dunno, I just think XML is garbage in general outside of the security realm too and got next to nothing right. I accept it's the only thing we could use for a system like XMPP, but IMO we should literally never use more of it ever regardless of whether it's one of the parts that has consistent security issues or not

  905. Zash

    asn1, the xml before xml, let's go back to it!

  906. Zash

    If only for the CRITICAL bit

  907. mathijs has joined

  908. Zash

    SamWhited, face it, everything we do is garbage held together with duct tape. That we haven't nuked/pandemic'd/burned ourselves into extinction yet is quite amazing. Probably because we invented duct tape 😀

  909. SamWhited

    you're not wrong

  910. arc

    That and because we are hanging just above the low hanging fruit. Like vulnerabilities in cryptocurrency exchanges that allow people to steal millions of dollars in untraceable funds

  911. StefanK$ has left

  912. Zash

    At least it's our garbage. Our own! Our ... precious!

  913. Zash pets pile of angle brackets.

  914. arc

    Sure people could attack xmpp. I imagine many of us could, knowing what we know. But there is no direct profit from doing so, and any scheme to profit from such an attack would be too complicated to pull off

  915. moparisthebest

    I think this can be summarized as "computers are bad, formats don't matter" https://duckduckgo.com/?q=json+parsing+vulnerabilities https://duckduckgo.com/?q=xml+parsing+vulnerabilities

  916. DebXWoody has left

  917. DebXWoody has joined

  918. Zash

    https://xkcd.com/2030/ comes to mind

  919. SamWhited

    That's exactly the problem, "our entire field is bad" doesn't mean "so we should give up and use whatever we want and ignore the problems with specific formats because another one might also have problems or there might be an issue at some other layer of the stack"

  920. moparisthebest

    but, switching formats doesn't address the problem, you should just address the problem instead

  921. Zash

    "our entire field" has existed for like half a century and usually doesn't directly kill people ... unless you count artillery trajectory calculations ... oh no

  922. fuana has joined

  923. SamWhited

    I didn't say we should switch formats, I said XML is what we've got and it's probably the only thing that works for XMPP but we shouldn't be dismissive when people point out problems with it and we shouldn't add more of it.

  924. Zash

    SamWhited, and we shouldn't do what everyone else does and wrap it in JSON!

  925. fuana has left

  926. fuana has joined

  927. Zash

    Oh wait, oh no, https://xmpp.org/extensions/xep-0295.html

  928. SamWhited

    I mean, yah, we should definitely not implement XEP-02395 :)

  929. SamWhited

    err, 0295

  930. APach has left

  931. APach has joined

  932. arc

    No Sam. We should fix it. Here I'll put it on my giant stack of things I care about that will never earn a cent, will not improve my chances of getting hired, would take the rest of all of our lifetimes, and leave us living on the street.

  933. fuana has left

  934. Kev

    Although 295 was obviously a joke, I note that the first suggested encoding pretty much works, and actually avoids some of the security issues of XML ;)

  935. Zash

    The one that looks a bit like JSON-LD?

  936. arc

    I do really care about these things. I'm not completely jaded. I think a lot of people really care about these things. I'm pointing out that we need to solve some lower-level fundamental problems first.

  937. Kev

    Zash? Does it? Yes, anyway. The one that isn’t entirely stupid.

  938. Kev

    When I wrote it I didn’t think of it in terms of having benefit over the XML representation, but it *does* eliminate whole classes of security issues.

  939. Andrzej has left

  940. adiaholic has left

  941. Daniel has left

  942. debacle has joined

  943. stpeter has joined

  944. stpeter has left

  945. Andrzej has joined

  946. fuana has joined

  947. fuana has left

  948. ti_gj06 has joined

  949. fuana has joined

  950. adiaholic has joined

  951. fuana has left

  952. Tim has left

  953. fuana has joined

  954. stpeter has joined

  955. stpeter has left

  956. fuana has left

  957. werdan has joined

  958. DebXWoody has left

  959. fuana has joined

  960. pasdesushi has left

  961. fuana has left

  962. werdan has left

  963. nyco has left

  964. Neustradamus has joined

  965. marek has left

  966. marek has joined

  967. adiaholic has left

  968. inky has left

  969. fuana has joined

  970. ben has left

  971. benharri has left

  972. Mikaela has left

  973. Neustradamus has left

  974. Neustradamus has joined

  975. DebXWoody has joined

  976. fuana has left

  977. Lance has joined

  978. adiaholic has joined

  979. papatutuwawa has left

  980. fuana has joined

  981. Tobias has left

  982. adiaholic has left

  983. andy has left

  984. fuana has left

  985. goffi has left

  986. andy has joined

  987. fuana has joined

  988. nyco has joined

  989. benharri has joined

  990. inky has joined

  991. fuana has left

  992. adiaholic has joined

  993. APach has left

  994. APach has joined

  995. APach has left

  996. APach has joined

  997. ben has joined

  998. x51 has left

  999. andy has left

  1000. alameyo has joined

  1001. Adi has left

  1002. Adi has joined

  1003. fuana has joined

  1004. fuana has left

  1005. pasdesushi has joined

  1006. pasdesushi has left

  1007. pasdesushi has joined

  1008. pasdesushi has left

  1009. pasdesushi has joined

  1010. pasdesushi has left

  1011. pasdesushi has joined

  1012. ti_gj06 has left

  1013. pasdesushi has left

  1014. pasdesushi has joined

  1015. pasdesushi has left

  1016. pasdesushi has joined

  1017. pasdesushi has left

  1018. pasdesushi has joined

  1019. pasdesushi has left

  1020. pasdesushi has joined

  1021. pasdesushi has left

  1022. pasdesushi has joined

  1023. andrey.g has joined

  1024. sonny has left

  1025. sonny has joined

  1026. sonny has left

  1027. pasdesushi has left

  1028. sonny has joined

  1029. jcbrand has left

  1030. sonny has left

  1031. sonny has joined

  1032. sonny has left

  1033. sonny has joined

  1034. alameyo has left

  1035. intosi has left

  1036. nyco has left

  1037. wurstsalat has left