XSF Discussion - 2019-12-13

  1. Shell has left

  2. Shell has joined

  3. kokonoe has joined

  4. !XSF_Martin has left

  5. stpeter has joined

  6. sjaak has left

  7. sjaak has joined

  8. sjaak has left

  9. sjaak has joined

  10. !XSF_Martin has joined

  11. larma has left

  12. larma has joined

  13. debacle has left

  14. SpaceFreak aka Tracer has joined

  15. mukt2 has joined

  16. sjaak has left

  17. sjaak has joined

  18. sjaak has left

  19. sjaak has joined

  20. sjaak has left

  21. sjaak has joined

  22. sjaak has left

  23. sjaak has joined

  24. mukt2 has left

  25. SpaceFreak aka Tracer


  26. SpaceFreak aka Tracer

    thats zero o'clock content

  27. sjaak has left

  28. sjaak has joined

  29. SpaceFreak aka Tracer

    :64 participants at this time is actually sick~

  30. eevvoor has left

  31. sjaak has left

  32. sjaak has joined

  33. sjaak has left

  34. sjaak has joined

  35. SpaceFreak aka Tracer has left

  36. calvin has joined

  37. mukt2 has joined

  38. stpeter has left

  39. andy has left

  40. stpeter has joined

  41. !XSF_Martin has left

  42. !XSF_Martin has joined

  43. moparisthebest

    Wonder how that works copyright-wise with nginx

  44. moparisthebest

    I'd hope if you don't enforce your copyright for 20 years while the product gains widespread adoption you don't get to swoop in and take it, but who knows

  45. mukt2 has left

  46. mukt2 has joined

  47. krauq has left

  48. kokonoe has left

  49. krauq has joined

  50. sjaak has left

  51. sjaak has joined

  52. sjaak has left

  53. sjaak has joined

  54. adiaholic has joined

  55. pdurbin has joined

  56. Arc

    well that didnt stop them from arresting the lead two developers

  57. Arc

    I assume their equivalent of a DA approved the arrest, usually required to get a justice to sign of

  58. mr.fister

    moparisthebest: that's the copyright law, destructive by design

  59. Arc

    but for a company that considers working on other FOSS projects to be a conflict of interest, I don't feel bad for them.

  60. Arc

    if they were good actors in the community I'd be happy to come to their defense

  61. !XSF_Martin has left

  62. !XSF_Martin has joined

  63. pdurbin has left

  64. Arc

    they were literally the last company I interviewed for in SF.

  65. Zash has left

  66. Arc

    the first interview was great. sure there were accent barriers but it seemed like a great fit. and then being introduced to their CEO who turned out to be a former rugby teammate from washington DC

  67. Zash has joined

  68. mukt2 has left

  69. Arc

    and that amazing high turned 180 when the CEO told me, point blank, that I'd have to resign from working on other FOSS projects while working for them because it'd be considered a conflict of interest ---

  70. kokonoe has joined

  71. arc has left

  72. arc has joined

  73. !XSF_Martin has left

  74. mimi89999 has left

  75. mimi89999 has joined

  76. moparisthebest

    Actually I'd heard you tell that story before so I didn't feel sorry for them either

  77. moparisthebest

    I was asking for more selfish reasons, like, can I keep using nginx :)

  78. sjaak has left

  79. sjaak has joined

  80. sjaak has left

  81. sjaak has joined

  82. mukt2 has joined

  83. Daniel has left

  84. Daniel has joined

  85. !XSF_Martin has joined

  86. mr.fister

    moparisthebest: you'll know it only after a while, the rambler group states that "any usage of nginx without their personal permission is violation of copyright law" or something like that

  87. waqas has left

  88. mukt2 has left

  89. !XSF_Martin has left

  90. UṣL has left

  91. Shell has left

  92. mukt2 has joined

  93. !XSF_Martin has joined

  94. calvin has left

  95. calvin has joined

  96. mukt2 has left

  97. adiaholic has left

  98. adiaholic has joined

  99. andrey.g has joined

  100. karoshi has joined

  101. mukt2 has joined

  102. krauq has left

  103. Zash has left

  104. krauq has joined

  105. calvin has left

  106. !XSF_Martin has left

  107. !XSF_Martin has joined

  108. matkor has left

  109. matkor has joined

  110. pdurbin has joined

  111. kokonoe has left

  112. kokonoe has joined

  113. pdurbin has left

  114. !XSF_Martin has left

  115. !XSF_Martin has joined

  116. mukt2 has left

  117. !XSF_Martin has left

  118. !XSF_Martin has joined

  119. Daniel has left

  120. Daniel has joined

  121. ralphm has left

  122. ralphm has joined

  123. Nekit has joined

  124. Yagiza has joined

  125. mukt2 has joined

  126. !XSF_Martin has left

  127. !XSF_Martin has joined

  128. ralphm has left

  129. ralphm has joined

  130. !XSF_Martin has left

  131. !XSF_Martin has joined

  132. stpeter has left

  133. andy has joined

  134. karoshi has left

  135. andy has left

  136. andy has joined

  137. neshtaxmpp has joined

  138. lorddavidiii has joined

  139. mukt2 has left

  140. stpeter has joined

  141. pdurbin has joined

  142. stpeter has left

  143. kokonoe has left

  144. adiaholic has left

  145. adiaholic has joined

  146. pdurbin has left

  147. mr.fister has left

  148. wurstsalat has joined

  149. Douglas Terabyte has left

  150. paul has joined

  151. DebXWoody has left

  152. stpeter has joined

  153. kokonoe has joined

  154. DebXWoody has joined

  155. pdurbin has joined

  156. stpeter has left

  157. Douglas Terabyte has joined

  158. pdurbin has left

  159. LNJ has joined

  160. !XSF_Martin has left

  161. Daniel

    > I assume their equivalent of a DA approved the arrest, usually required to get a justice to sign of Arc: it's Russia

  162. stpeter has joined

  163. neshtaxmpp has left

  164. stpeter has left

  165. !XSF_Martin has joined

  166. mukt2 has joined

  167. LNJ has left

  168. mukt2 has left

  169. sonny has joined

  170. kokonoe has left

  171. kokonoe has joined

  172. arc has left

  173. arc has joined

  174. Arc

    certainly its Russia. :-)

  175. moparisthebest has left

  176. rion has joined

  177. Tobias has joined

  178. karoshi has joined

  179. Arc

    I don't expect their former employer will win. But the drama will likely drag out for some time, especially influence their current round of VC funding, and will hurt their user base

  180. !XSF_Martin has left

  181. !XSF_Martin has joined

  182. matkor has left

  183. matkor has joined

  184. UṣL has joined

  185. SpaceFreak aka Tracer has joined

  186. sonny has left

  187. adiaholic has left

  188. Arc

    its like Michael Holdmann trying to patent parts of XMPP and then use those patents as collateral for VC loans. Its nasty behavior that can only hurt the community when his business fails and those patents get sold to a patent troll

  189. adiaholic has joined

  190. Arc

    if NGINX Inc never existed, if nginx remained a FOSS project, or tried to commercialize without VC funding or trying to turn themselves into a FOSS-toxic tech startup, we wouldn't be where we are now. but now that they have, the VC money they've pocketed makes them an easy target for copyright trolls.

  191. sjaak has left

  192. sjaak has joined

  193. SpaceFreak aka Tracer has left

  194. !XSF_Martin has left

  195. !XSF_Martin has joined

  196. mathijs has left

  197. mathijs has joined

  198. sonny has joined

  199. SpaceFreak aka Tracer has joined

  200. krauq has left

  201. krauq has joined

  202. dwd

    I think I'm broadly in agreement with lovetox here. I think Kev's proposal on list goes some way to addressing these concerns, and we'll work on a concrete proposal in this space next week.

  203. SpaceFreak aka Tracer has left

  204. nyco

    thread regarding IoT : https://fosstodon.org/web/statuses/103293746877193364

  205. adiaholic has left

  206. Arc

    nyco: XEP 0322 needs some serious work or should be killed. its in a non-starter state because it was written by someone with only the most cursory experience with EXI

  207. nyco

    yep, maybe we should reboot the IoT stuff

  208. Arc

    i dont expect to see any forward movement from that from the xmpp council because EXI is very much not simple

  209. Arc

    grammar distribution and negotiation needs to be solved.

  210. Arc

    and yes while grammar unaware mode is available, and could be used, it would dramatically increase the software size for embedded devices and thus make it not feasible for some iot cases

  211. Arc

    I think I might be the only current XSF member with any experience working with EXI and that would need to change to move forward with it

  212. nyco

    there is some more members with EXI experience, not judging on how much

  213. Arc


  214. nyco

    yeah, indeed, EXI may not be the way to go

  215. nyco

    Arc MongooseIM guys

  216. nyco

    at least

  217. adiaholic has joined

  218. Arc

    ive never gotten that impression

  219. Daniel

    On the topic of exi. I was in a meeting yesterday in which we essentially decided to reinvent exi because exi itself was deemed to complicated

  220. pdurbin has joined

  221. Daniel

    But we do a strong need for 'something like exi'

  222. Arc

    its not too complicated, its poorly documented.

  223. Arc

    EXI itself is incredibly simple

  224. Daniel

    Meaning compression with predefined dictionary

  225. Arc

    the string dictionary?

  226. Daniel

    A xml tag name namespace tuples dictionary

  227. Daniel

    So yes, I guess

  228. Arc

    yea so that's really simple. too simple.

  229. Arc

    the only place so far I've had EXI completely fail is SVG

  230. Arc

    but the problem with the way its documented is it feels ass backwards. we tend to view XML from a data -> in perspective, because that's the way virtually every piece of software is written

  231. beta has left

  232. beta has joined

  233. Arc

    its best to view EXI from the viewpoint of the data consumer; typically from the viewpoint of that critical piece of code that does a string compare to match it with code to be executed when a certain element is found

  234. Arc

    in XMPP code, that means the stanza router

  235. mathijs has left

  236. mathijs has joined

  237. Arc

    somewhere in virtually every piece of XMPP software I've ever seen is a string compare routing an incoming stanza for message, iq, presence, or various stream errors/etc.

  238. Zash has joined

  239. Arc

    in EXI you always have the grammar - whether its schema informed strict (whereas grammar is fixed), schema informed non-strict (whereas grammar can be extended), or schema uninformed (whereas the grammar is constructed as needed).

  240. dwd

    Arc, But string compares only after parsing, right?

  241. neshtaxmpp has joined

  242. Daniel

    I guess what I'm saying is that we have a need for some form of binary xml. But exi feels too much for us to tackle alone. (as opposed to just coming up with something new). But if someone else wants to colab on an exi 2.0 xep or something I might be able to convince my people to do that instead

  243. Arc

    dwd in traditional XML sure, string compares after layer 2 parsing.

  244. dwd

    Arc, But EXI prevents this?

  245. Arc

    in EXI you should only do string compare when parsing the grammar.

  246. dwd

    Arc, I think I need an example, I'm being stupid.

  247. Arc

    your parser isnt receiving the string "message" anymore. its receiving the integer 1 instead. the grammer determines how many integers there are at a given scope, and what those integers mean

  248. Arc

    ok. say we're in the scope of <stream:stream> and lets ignore stream management and stream:error etc; we have 4 possible options in xmpp; message, iq, presence, and </steam:stream>

  249. Arc

    EXI understands those as 0, 1, 2, and 3.

  250. Arc

    when in bit-packed mode EXI will represent those options with 2 bits. in other modes as one byte. but in parsing it doesn't matter

  251. pdurbin has left

  252. Arc

    on decompression you set a hook for message, iq, presence, and </stream:stream> like you always have. the difference is now you're not string comparing what the XML library is giving you. your code is being called on a switch block for 0, 1, 2, and 3 respectively. which is tremendously faster

  253. Arc

    for embedded IoT devices using strict schema-informed grammar you can hard code that so that 0 is always message, 1 is always iq, 2 is always presence, and 3 is always </stream:stream>

  254. Arc

    so you're decoding a <iq> element, in that scope there's be a fixed number of attributes and child elements that are understood by the grammar, and similarly each is represented by an integer. with strict schema-informed grammar those options will always be the same. so a byte 1 followed by a byte 2 may always be "<iq from="

  255. stpeter has joined

  256. Arc

    the only time a string compare should ever be needed is when transmitting grammar or grammar extensions for unexpected things that arrive, and then only once to see if your code has any clue what those things do.

  257. kokonoe has left

  258. LNJ has joined

  259. Arc

    after that you'll only use the integer for that thing, and you can record a pointer to what to execute what to do when that thing is read.

  260. !XSF_Martin has left

  261. !XSF_Martin has joined

  262. Arc

    the XMPP challenge for the XEP is for how the client informs the server what grammer to use. strictly "schema uninformed" would waste a lot of bandwidth and consequently battery power on the IoT device. ideally we would like the device to not have to transfer the grammer also to save power. eg it could come from the manufacturer

  263. LNJ has left

  264. Arc

    there's two major problems with XEP 0322; first its relying on "compiling" a schema from a given set of schema files (schema informs grammar, but lacks any definition for the precise way to do this reliably across different libraries), and second its using arbitrary URLs to load the schema files from which is a pretty major security issue

  265. Arc

    clients should not be able to cause a server to automatically download HTTP URLs; that could be used as part of a DDoS attack, to anonymously transmit data via some web APIs, or worse

  266. sjaak has left

  267. sjaak has joined

  268. marc_ has left

  269. Arc

    when I first started looking at EXI 5+ years ago I thought strict grammar mode would be unusable in any real world case. but the more you work with it, the more it changes your way of seeing XML, and you start to see strict grammar as the obvious default. there's certainly cases for extensible grammar but in all seriousness, software is written to process what the software is written to process.

  270. kokonoe has joined

  271. Arc

    that is to say: unless you're debugging or planning to save XML for other software to process, there's no reason to receive XML elements or attributes that the software is going to ignore anyway. its better for the server to silently drop that content instead of sending it to the client where it'll be ignored because there's the client doesn't know how to process it.

  272. marc_ has joined

  273. stpeter has left

  274. mukt2 has joined

  275. Daniel

    Fwiw I'd be fine with a potential exi 2.0 xep being only specified for strict

  276. intosi has joined

  277. Daniel

    That might limit exi use to closed deployments but that's where it is needed the most

  278. Arc

    well one thing the XEP does right IMHO is the client dictates the grammar. so the client decides.

  279. Arc

    Ive been able to compile libexi for one specific small grammar in under 1k of rom. and that's the point of it for IoT.

  280. Arc

    but there's really little need to restrict what the server can support. any xmpp server should be able to support all the various modes. plus - even with strict grammar, the client still needs a way to inform the server what the grammar is.

  281. Arc

    when your IoT device is monitoring air pollution while attached to the top of a city bus with rare earth magnet and powered by a solar cell, or monitoring soil moisture in your garden with a single AA battery, the transmit power (plus amount of time the CPU is out of sleep mode) to send the grammar is simple too much.

  282. Arc

    to convince a lot of IoT companies to change to XMPP we need to meet or beat the power use of their current protocol. which in most cases will look pretty similar

  283. dwd

    So my use for EXI would be handling the crappy bandwidth one gets in hospitals. I don't think I'm anywhere close to XMPP being the bandwidth hog here, mind, but still. I don't understand how a schema-strict mode would work (or, at least, how it wouldn't violate the end-to-end principle and all that implies).

  284. Arc

    schema-strict mode would do the exact same thing that current text XML does, just in a different place. if your current client ignores data it doesn't want or need, the server would drop that data instead.

  285. Arc

    and really, for most extensions, its a best practice not to send data that the receiver hasn't asked for or specified support for

  286. Guus

    Hey, Arc!

  287. Arc

    hey Guus!

  288. Guus

    Longtime no see. How are you? (maaaybe I should've waited for whatever conversation was going on to finish, sorry 😉 )

  289. dwd

    Also, yes, it's great to see Arc back.

  290. Arc

    dwd but also for hospital IoT stuff, they likely have the memory and CPU for handling non-strict schema. for the strict schema, its mostly for tiniest of IoT devices.

  291. Arc

    thanks! :-)

  292. dwd

    Arc, We're not IoT, but yes.

  293. Arc

    yea so informed non-strict should be fine.

  294. mimi89999 has left

  295. mukt2 has left

  296. Arc

    there's absolutely an overhead for building grammar tables that may be extended. I have a device within arms reach of me right now that's running a 8-bit RISCV that it would be much more difficult to squeeze extensible tables into.

  297. mimi89999 has joined

  298. Arc

    (its not *really* 8-bit but that's another topic)

  299. Ge0rG

    Arc: would it make sense to have a grammar caching mechanism similar to entity caps?

  300. Guus

    Daniel My first gut reaction to trying to create a new something something that's EXI like but not EXI is https://xkcd.com/927/ - but knowing you, you've probably considered that. I'm interested in how a new approach is different in this way.

  301. Ge0rG

    The client transmits its grammar on first g connection, it's cached on the server, problem solved

  302. Ge0rG

    The client transmits its grammar on first connection, it's cached on the server, problem solved

  303. Ge0rG

    On the second connection, the client just sends the sha256 of the grammar, no need to retransmit

  304. Ge0rG

    When the client software is updated, the new hash isn't known to the server, it requests the new grammar

  305. Ge0rG

    The server side filtering to known elements would also be good for mobile clients, not just IoT

  306. Guus

    If I understand Arc correctly, then one of primary issues is that EXI is lacking proper documentation. How much effort would it take to get that fixed? Would we be able to find sponsoring for that, given the impact it could have on certain applications?

  307. Ge0rG

    We really need to position ourselves as The Solution for IoT

  308. Arc

    Guus: its not a lack of documentation, its that the docs are terrible and overly verbose, such that its really difficult for people approaching EXI to understand it.

  309. Guus

    (From what I'm reading just here, at least Arc, Daniel and Dave are dabbling in this field. Is there some corporate backing here?)

  310. Ge0rG

    Arc: you could write a book!

  311. Arc

    Ge0rG: that is exactly something I proposed a few years ago, and I now disagree with as an approach. the problem is you don't want to burden the client with that responsibility

  312. Guus

    Arc sure. "documenation needs work."

  313. Guus

    Arc sure. "documentation needs work."

  314. dwd

    Guus, We aren't dabbling at all. My guess would be that Kev (And Isode, more generally) would be more interested than us.

  315. dwd

    Guus, I'm just EXI-curious. :-)

  316. Guus


  317. Guus

    You're talking about the extremely low-bandwidth type of applications?

  318. Ge0rG

    Arc: you also said you don't want the grammar to be an external HTTP URL. What's your proposal?

  319. SpaceFreak aka Tracer has joined

  320. Guus

    You're talking about the extremely low-bandwidth type of applications that some of their customers are working with?

  321. Arc

    Guus: not just low bandwidth, but low handware, low cpu, low ram, low battery power. my ultimate goal of EXI isn't to save bandwidth but vastly expand XMPP into the embedded market

  322. Daniel

    Guus: I was in fact coming from a "let's consider exi" position. Precisely not to reinvent the wheel and to have something that benefits the community. I think the feeling on the other side of the table was that "something like funxmpp" (essential hard coded, preshared lookup table of commonly used elements) is easier to grasp

  323. dwd

    Guus, Right. Most of our problems are with RTTs and unstable networks - these also apply to Isode's space. Bandwidth isn't great for us, but it's waaay better than some of Isode's cases. Our clients run on consumer smartphones, so we're absolutely not an IoT case.

  324. Daniel

    Than exi with its 3 modes of operations

  325. Daniel

    The fact that I don't know a lot about exi didn't help

  326. Arc

    Ge0rG: that's something to be discussed. I mean a specific XMPP extension for sending a grammar over XMPP would potentially be doable. that way it could never be used for an attack.

  327. Guus

    Daniel that's fair. But also kind of boils down to the documenation problem with EXI, right?

  328. Daniel

    But my hope was that starting a broader effort on the exi front might help me convince the other side

  329. Daniel

    Yeah. Also that there are virtually no libraries

  330. Arc

    ok sorry as soon as i wrote that last bit I realized how stupid that is. literally anything could be formed into an attack vector by someone determined enough. the local antifa group took down a hate group using craigslist ads.

  331. Guus

    Arc I understand. I was trying to describe to Dave the customers that I thought were interested in this, without naming names. I wasn't trying to describe all of EXI's benefit.

  332. Daniel

    I mean if exi was some plug and play thing that you just hook in between things it might be easier

  333. Arc

    Daniel: yes, and lack of software is absolutely a problem I want to address. libexi is my strategy for that but there's zero funding for it

  334. Ge0rG

    Arc: the alternative would be a common grammar file format and a central repository, which your server needs to pull from whenever you launch a new version client

  335. Ge0rG

    Arc: really, I don't see a viable alternative to xmpp based delivery for anything but closed deployments

  336. Arc

    Ge0rG: yea a "registry" could work.

  337. Ge0rG

    Arc: but server operators are lazy, and automatic push + pull with a central registry is even worse, security wise

  338. Arc

    that's something XSF could maintain

  339. Guus

    I'm probably the least qualified / experienced of people talking here right now, but from what I read here is that EXI has already had quite some work gone into it, but needs some work to be (re)usable (docs and libs). Feels like a shame to replace it with an easier-to-develop (but really, that rarely turns out to be the case) yet less re-usable XMPP-specific solution.

  340. !XSF_Martin has left

  341. !XSF_Martin has joined

  342. dwd

    Guus, I'll fight you for the least-qualified position. ANd probably lose.

  343. Arc

    Guus: yes and no. in Java-land EXI is extremely mature. in C-land, not so much. outside of Java there's simple nobody working on XML.

  344. Guus

    Would it be good to get some people together, and see what amount of resources would be needed to use EXI to get a real-world application, that we can pitch to customers.

  345. Arc

    there's known 0-days in libxml2 that could easily cost billions in damage and nobody working on it.

  346. Guus

    Arc well, we have several Java implementations, so that might be a good platform to do some prototyping in XMPP then.

  347. Guus

    If anything, that could lead to traction.

  348. dwd

    Arc, Presumably strict-schema mode in EXI would require stricter schemas (ie, entirely immutable schemas) than we have previously used in XMPP?

  349. Arc

    Google knows about them, so they wrote their own XML library in Go. and Mozilla wrote their own in Rust for servo, which is likely better

  350. dwd

    Arc, Or can we safely add further attributes with default values (for example) without having to ... recompile? ... the schemas for EXI?

  351. Arc

    dwd: strict grammar means that any unknown elements, attributes, etc be ignored.

  352. Guus

    (Which seems to be problematic for the "X" in XMPP?)

  353. Arc

    Guus: no, it does the same thing in most cases when an unknown element or attribute is received by software. the client you use now for example, if I sent <message ...> <helloworld/><body>hi</body></message> what would it do with the <helloworld/> ?

  354. goffi has joined

  355. Steve Kille has left

  356. Guus


  357. Arc


  358. dwd

    Yes, I was thinking it's not a terrible thing, since it's how XMPP works itself.

  359. Arc

    it just ignores it on the server. another way of looking at EXI is, its as if part of the XML parsing gets moved to the server

  360. Guus

    but would EXI still pass along the data, which would then be parseble XML?

  361. Arc

    not if strict is sent.

  362. Arc

    its a bitflag in the EXI header

  363. Steve Kille has joined

  364. dwd

    Arc, But the, erm, EXI schema definitions need to be the same on both server and client?

  365. Guus

    So, if Client A wants to send Client B some data that contains data not understood by the server, how'd that work?

  366. Arc

    dwd the grammar needs to be the same, yes. the schema informs the grammar.

  367. Arc

    Guus: if you send me a <message><helloworld/> and I'm connected to my server while specifying "strict" mode, then my server will drop <helloworld/>

  368. Guus

    If client A sends stuff not understood by the server, which drops data, client B is unlikely to receive that data?

  369. Arc

    here's what the EXI header looks like, which might be the most readable part of the whole specification

  370. Arc


  371. Guus

    Isn't that wildly different as compared to regular XMPP? If a server receives something it doesn't recognize, it'll ignore it, but pass it along.

  372. dwd

    Guus, Exactly that. Which is why I say it conflicts with the end to end principle.

  373. Arc

    Guus: the client dictates what grammar to use, not the server.

  374. Guus

    if EXI doesn't do that...

  375. Steve Kille has left

  376. Kev has left

  377. !XSF_Martin has left

  378. !XSF_Martin has joined

  379. Guus

    ah ok - so if the client is going to send something, it will (well, _can_) always inform (and thus configure) the receiving end in such a way that it can properly parse all data?

  380. Arc


  381. Guus

    Right. And the receiving client will have configured the server in a similar way, so that it can be send all data that it'll be able to interpret.

  382. Zash

    So the server still needs to shuffle some generic XML representation internally?

  383. Zash

    Can an EXI client even send something not part of the schema/grammar?

  384. Guus

    (Can that grammar be re-negotiated? For example when new plugin gets dynamically loaded that introduces new goodies)

  385. Arc

    Zash: yes there's really no way around that because the clients, in most cases, will be speaking different grammars. and no the client cannot encode new things in strict encoding.

  386. Arc

    Guus: I don't believe so, the "schemaId" gets sent as part of the EXI header. though the XEP could include a specific stream-level message to renegotiate the EXI?

  387. Guus

    Somewhat of an edge case

  388. Guus

    dwd with the above, does your concern regarding end to end principle hold?

  389. Guus

    as with Arc explanaition, I'm thinking that it at least _migth_ work 🙂

  390. Arc

    yea in those cases I think reconnecting is perfectly fine. but there is the "fragment" option in the EXI header too, such that you're specifying that you're not sending the whole XML document from <root> to </root> but only part of it.

  391. dwd

    Guus, Well, I said it conflicts, and it does, but that's not a blocker.

  392. Guus

    how does it conflict?

  393. Arc

    in that case yes even without an XMPP stream-level support, if you wanted to change options on the fly, you could. with obvious security concern of trying to DDoS a server by flooding it with different schemaId's to be retrieved from a ... "grammar registry"

  394. Guus

    client A won't be able to send client B data that client B doesn't know how to parse. That?

  395. Guus

    Arc I'm unsure if those conerns are obvious, but sure 🙂

  396. dwd

    The end-to-end principle essentially says that end-clients care about more layers than intermediate routers. That's always been prtty shaky in XMPP anyway, since the server is often an end-client in that sense.

  397. sjaak has left

  398. sjaak has joined

  399. Guus

    That's way to abstract for me to understand without loads more coffee.

  400. Arc

    what about being able to block JIDs? I mean, the client is literally instructing the server to drop unknown content, that's not unlike instructing the server to not send content from certain JIDs?

  401. Guus

    (but you said 'no blocker' which I'm happy to accept)

  402. Zash

    To the coffee machine!

  403. Arc

    wow its 2am here.

  404. dwd

    Arc, It's armageddon here, for comparison.

  405. Guus

    At least there's clarity... ?

  406. dwd

    Yes, our doom can be seen very clearly.

  407. Arc

    oh yea that's right, I completely forgot I jumped back in xsf to ask for recommendations for a xmpp library with pubsub support for rust

  408. sjaak has left

  409. sjaak has joined

  410. Guus

    Dutch weather is as miserable as yours this time of year. We'd be happy to have you 🙂

  411. dwd

    Arc, Rust... I'm not sure there's a stand-out library for Rust. I'd be interested to see one.

  412. Arc

    i recognized a few names from the cargo package list

  413. dwd

    Guus, I have given it some thought. Not much less convenient for travel to the office. But it'd mean moving my daughter's schooling at this point, which'd be tricky.

  414. Guus

    Isn't that why you learned the other kid to fly? Commute solved.

  415. lorddavidiii has left

  416. Guus

    So, if we want to (and there seems interest, judging by this conversation) want to move forward with EXI in XMPP world, what would we need? It might be good to come up with an estimate of the effort involved, to be able to attract sponsorship for this.

  417. dwd

    And his license is an EU one, so covers him. No, I've genuinely thought about this.

  418. Arc

    Guus: I would really appreciate having a working group, even an informal one.

  419. Guus

    dwd it's a sucky situation. At least now, things will need to be solved pragmatically. Hopefully, things turn out less bad than what you've now envision.

  420. Guus

    And face it, it could be worse: Farage at no 10?

  421. kokonoe has left

  422. Guus


  423. Guus

    Arc we can do that easily. Who'd be willing to dedicate their effort into that?

  424. Arc

    beyond the big issues like how to communicate grammar, there's a ton of small ones which need to be determined. like a standard set of fidelity options for XMPP. the spec is not a small job.

  425. Arc


  426. Zash

    Schemas being non-normative is also fun I imagine

  427. dwd

    Guus, https://twitter.com/DwdDave/status/1205426678639579136 - I'm desperately trying to find some positive outcomes.

  428. Arc

    Zash: yea. well that's also the difference between schema and grammar :-) schema informs grammar, but grammar is the actual bytestream format

  429. Arc

    schema cannot, by itself, construct a grammar. grammar is weighted, so unless you have some very cool code monitoring a connection and creating an optimized grammar for you, a human will actually write the grammar based on the schema

  430. Arc


  431. Guus

    Arc fwiw, my preferred way of working would be to _first_ come up with a proposal on how to get EXI in XMPP realized, with estimated effort requirements, budgets, etc, before digging into the actual implementation details. My goal would be to first secure funding.

  432. Guus

    Or: have the information needed to secure funding.

  433. Arc

    you have a sponsor in mind to fund such work? :-)

  434. Guus


  435. Guus

    but neither will bite without a proper plan.

  436. Arc

    that's excellent. I can't get companies here, even those already using XMPP, to bite.

  437. Guus

    Things should ideally be as concrete as: for this amount of money, you get this feature.

  438. Arc


  439. larma has left

  440. Guus

    So, how do we get to there?

  441. Guus

    (assuming we want to spend effort on this)

  442. Arc

    well ive got 5+ years implementation experience on EXI but I'm not a working group.

  443. Guus

    What would you need from a working group?

  444. Arc

    2 other people willing to invest the brain space to understand this 96 page technical document

  445. Guus

    (we can ask Board to make you a Working Group, but I'm sensing that that's not what you're after 😛 )

  446. Arc

    its not the official-ness of the group im concerned with, its the brains.

  447. Guus

    Yeah, that's what I was trying to get at.

  448. Guus

    any takers?

  449. Guus eyes room

  450. Guus

    it's less than 100 pages people!

  451. Guus ducks.

  452. Arc

    I'm happy to help people get up to speed. and honestly most of it is just a few pages in the EXI header specs, and understanding the grammar

  453. dwd

    I'm not sure I can commit to working hihg-intensity on this.

  454. larma has joined

  455. Arc

    there's also this super cute (and not terribly useful) primer https://www.w3.org/TR/exi-primer/

  456. Guus

    I personally won't commit unless I can find funding for it.

  457. kokonoe has joined

  458. Arc

    yea ive been without funding on this for all this time. I've met with a few companies and received some useful feedback but everyone just wants it for free.

  459. Arc

    though I learned Google has their own internal implementation written in Go

  460. Guus

    I admire you for doing that. It's not an investment I'd want to make myself.

  461. Arc

    apparently they already use it for push notifications?

  462. Arc

    I'm working on the premise of future marketability for libexi, given there's really nothing like it in existence. AGPL license it, and offer commercial LGPL-like exceptions, akin to wolfssl

  463. sjaak has left

  464. sjaak has joined

  465. Guus

    Do you want to drive an effort to find people for an XSF work team for this, Arc ?

  466. Arc

    yea but it might be from people not currently in the XSF

  467. dwd

    Arc, Sounds even better.

  468. Arc

    we dont have a ton of java people and that's really where we need. i'm absolutely not a java person

  469. Guus

    XSF Bylaws: "Participation in Teams shall be limited to elected Members of the Corporation."

  470. Guus

    <-- java

  471. Arc

    Q1 2020 membership page is open

  472. dwd

    Right, Guus is the most Java person I know.

  473. dwd

    Guus, You work around this by having a Work Team preside over a Special Interest Group, the latter being open to anyone.

  474. dwd

    Guus, Then the WOrk Team is basically devoted to getting people to understand and work with the process.

  475. Guus

    You like burning hoops and making people jump through them, don't you?

  476. Guus

    but, whatever. We will find some kind of venue to discuss this, one way or the other.

  477. Arc

    Guus: well since you're a java person :-) :-) :-)

  478. sjaak has left

  479. sjaak has joined

  480. Guus

    Arc I'll be as clear / blunt as Dutchies are known for: I'm definitely interested, but won't commit large amounts of effort without funding. If it boils down to doing this, or paid work, I'll go for paid work. I'm totally open to put in time to come up with data that can be used to secure funding.

  481. Arc

    I'm in the same state. Rent needs to be paid.

  482. Guus

    Hence my preferred order of things: first, do only what's needed to come up with budget estimates.

  483. Guus

    It would be awesome of others want to work under less strict conditions, but I personally choose not to.

  484. Arc

    certainly. libexi is something I've considered a hobby for now, and will continue to.

  485. Guus

    My problem is that I don't even have a grasp of the amount of effort that's required to get this to even a prototype state.

  486. Arc

    are you talking a spec or working code?

  487. Guus

    working code. My customers are unlikely to be interested in just a spec.

  488. Guus

    "for x amount of USD, you get EXI benefit A, B and C"

  489. Guus

    I'm not sure if that's the best first goal for us to work on, but I feel that that'd give me something I can use to try and secure funding.

  490. Arc

    the spec is going to end up being 20-30 pages. some working code in Java using an existing Java libraries? once the spec is done.. I don't imagine that being very long.

  491. Guus

    If others have another approach, I'd love to hear about that. I'm just winging it here.

  492. Arc

    from my side, a lot of my work has been in not just implementing EXI but using grammar to generate source code, and breaking the code up such that you can compile it with different options removed to save space

  493. Arc

    but thats all rust/C

  494. Arc

    and splatterings of assembly

  495. Guus


  496. Arc

    mozilla just recently declined sponsoring the work which would have gone into Servo. :-l but they made sure to jab the knife in my gut with the typical line "if you release it in an MPL compatible license we'd be happy to support it in Servo"

  497. stpeter has joined

  498. debacle has joined

  499. Guus

    yeah, I bet they would.

  500. Daniel has left

  501. Daniel has joined

  502. Guus

    I've tried to boil down what I'd like to know, before formulating a question to my customers. It's stupidly simple, but I find that that often helps coming up with concrete plans: https://etherpad.net/p/exi

  503. Guus

    I'd like that to be filled out.

  504. Arc

    at least they didn't drag me along with a job interview like a few companies have before telling me how they'll "love to support me by using my freely licensed software". one company asked for an hour presentation for their protocol team on EXI as part of the "interview" before telling me they'll join my "cause" if I change the license to permissive.

  505. Guus

    But I lack the knowledge to do that myself. Is this somethign we could work on?

  506. Arc

    I could contribute to it.

  507. Arc

    you want to find a Java EXI library and run some performance tests on it?

  508. Arc

    I know basic network performance for XMPP

  509. Arc

    I know that you *wont* get the 40x gains that I'm seeing from Java but you'll likely get 20x speed gains

  510. Arc

    the speed gains are largely the CPU cost of a string comparison. W3C did their own but I don't know what software they based it on, failing anything else you could grab some graphics from the EXI primer

  511. Arc

    er https://www.w3.org/TR/exi-measurements/

  512. Guus

    I'm updating that etherpad with the rough estimates that you have. I don't intend to send what is typed there verbatim. I'm fine with rough outlines, for now

  513. Arc

    outside IoT yea those speedups are the largest perk. especially on the server side.

  514. Guus

    those would be speedups in XML parsing right?

  515. Arc

    bandwidth and CPU cost money. saving bandwidth and CPU saves money.

  516. Guus

    what about bandwidth savings?

  517. Arc

    both parsing and creation, but yes. both sides

  518. Arc

    it depends on the data in all cases. IoT with float and int data obviously gets the best gains.

  519. Guus

    regular chat

  520. Guus

    ballpark it

  521. Zash

    "funxmpp"-like approach would get you bandwidth savings at the cost of CPU I imagine?

  522. Daniel

    > "funxmpp"-like approach would get you bandwidth savings at the cost of CPU I imagine? Probably. But CPU isn't our bottle neck at the moment

  523. stpeter has left

  524. Arc

    regular chat? you'll be using byte packed mode without the precompression option, qname tables under 250 in size, so "message" would be 1 byte, "from" is one byte. "to" is one byte. the JIDs would likely be one byte each. so SE-"Message" AT="from" "guus@yourhost.isgreat" AT="to" "arc@myhost.islong" ...message... EE would be around 60 bytes to around 6 bytes. so the message stanza overhead would be only around 10%

  525. Arc

    CPU processing speed, well see the measurements for the various java tools

  526. Arc

    with byte packed, you can assume that the qname for each element and attribute is 1 byte. and the string table for JIDs are 1 byte too. extra attributes like id= will add to the cost. and unique strings... if memory serves you can set the value for some attributes to not be cached since they're unique or not commonly reused. so the length of the id= attribute would be unchanged

  527. Guus

    right, but 'regular chat' traffic also includes a bunch of IQ and Presence stanzas. I'm thinking that most of the data would be pretty dictionairy-able, with the exception of the chat message texts?

  528. Arc


  529. Arc

    that's why i said overhead :-)

  530. Guus

    average stanza character count in my MAM archive appears to be 185

  531. Arc

    ok and whats the average body length for those message stanzas?

  532. !XSF_Martin has left

  533. Arc

    i mean i forgot the id= and the body element, so you have SE-"body", etc. but ballpark its a huge bandwidth saving

  534. Guus

    average body char count is 64

  535. Guus

    note that I'm doing stupid SQL queries on a bucket load of crappy data

  536. !XSF_Martin has joined

  537. larma

    And even if CPU isn't a bottleneck right now, saving on CPU is a great thing, because it reduces energy consumption on portable devices

  538. Guus

    so, what's the huge ballpark saving? 🙂

  539. Arc

    so average message header is 121, and we can reasonably say that can be represented in under 12 bytes.

  540. Arc

    so the message header would be 10x smaller. or the entirety of the message about 40% smaller.

  541. Arc

    but iq and presence nodes would reduce that further

  542. Arc

    ive never worked with that server but my own experience says that you should be able to double the number of stanzas routed per second. and thats exceedingly conservative

  543. Arc

    its likely closer to 10x

  544. Guus

    assuming that parsing is the bottleneck

  545. dwd

    Parsing is almost always the bottleneck.

  546. Arc

    yea you can't control things like disk or network. but parsing, most notably string matching, is usually the bottleneck

  547. Arc

    JIDs are being passed with a specific ID# instead of a string, so if done well you can optimize the crap out of that.

  548. Arc

    but even ignoring JIDs you've got the qnames. and that's a big part of it too

  549. Zash

    Isn't that a security problem?

  550. Arc

    not really, the JIDs to ID# is per connection only. they're just cached.

  551. Guus

    Cool. Updated my EXI-budget-for-Dummies at https://etherpad.net/p/exi

  552. Guus

    What work do we need to do for this?

  553. Arc

    https://www.w3.org/TR/exi/#encodingOptimizedForMisses details it but very basically, the first time "zash@ishome" is sent its cached at a specific lookup ID, and then that ID is used on that specific stream

  554. Arc

    you already have java libraries implementing EXI so your work is much simpler. most of the time will be on the XEP.

  555. Guus

    we'd probably want at least one client library too

  556. Guus

    Smack is Java...

  557. Arc

    sure. smack sounds good. ideally with a different library.

  558. Arc

    there's a few javascript libraries too

  559. Arc

    last I looked they all implement only the core functionality and you need to provide the grammar as a file

  560. Arc

    ((that's fine for a client))

  561. Arc

    i mean, for the client you shouldnt even need a library. you can hardcode values.

  562. Guus

    (note that I'm basing this very specific though experiment on implementations that I'm familiar with - I'm not trying to hijack this for my implementations at all, even if it starts to look like that now)

  563. Arc

    no you're fine. by no means would i want to intermangle my work into this

  564. Guus

    I'm also looking around the room

  565. Arc

    i made the mistake of leaking the source for libexi a few years ago and I still get email in korean asking for support

  566. Guus

    is EXI suitable for federation?

  567. sonny has left

  568. Guus

    client A -> server A -> server B -> client B ... even if client A and B share a vocabulary, servers A and B might not share the same?

  569. Arc

    S2S? absolutely. with a very broad grammar and without "strict" so that grammar can grow

  570. Guus

    what's the implication of 'strict' exactly? More performance?

  571. Arc

    non-strict... in XMPP terms, say you have four options for the scope inside <stream:stream>; <message>, <iq>, <presence>, and </stream:stream> - right?

  572. Arc

    strict would encode that as 4 options, or when bitpacked 2 bits

  573. Arc

    non-strict would encode that as 5 options with the final option being "something I didn't expect"

  574. Guus

    right, so, performance.

  575. Guus

    you want strict where possible, because it compresses better.

  576. Guus

    I need to be paying attention to other stuff shortly.

  577. Arc

    yes, minorly for bandwidth, more for CPU.

  578. Arc

    its 3:30am here so I really need to be doing other stuff shortly. ie, becoming unconsciousness and hopefully hallucinating vividly

  579. Guus

    Right. I like having been able to come up with the roughest of sketches.

  580. Guus

    Let's see if we can build on this later.

  581. Arc


  582. !XSF_Martin has left

  583. Arc

    feels like progress though, that's really great :-)

  584. !XSF_Martin has joined

  585. Tobias has left

  586. Guus


  587. Guus

    ok, thanks!

  588. Guus

    I'll grab some lunch and get other stuff done

  589. Guus

    sleep tight!

  590. Arc

    I was just re-reading https://www.w3.org/TR/exi-primer/#encoding and its actually .. a lot better than I remember.

  591. Arc

    i mean, wall of text is intimidating but if you walk through it and compare to the binary below its very informative, if cumbersome

  592. Arc

    but yea, g'night

  593. Tobias has joined

  594. pep.

    "Arc> oh yea that's right, I completely forgot I jumped back in xsf to ask for recommendations for a xmpp library with pubsub support for rust" < We have no funding, but there's xmpp-rs (xmpp:chat@xmpp.rs?join), You might be able to reuse bits and pieces, and otherwise we'd be happy to have a use-case (and people contributing)

  595. Wojtek has joined

  596. arc has left

  597. arc has joined

  598. pep.

    I don't think there's many more people working on Rust publicly atm, it'd be a shame to split already.

  599. arc has left

  600. arc has joined

  601. pep.

    dwd, ^

  602. Shell has joined

  603. pdurbin has joined

  604. dwd

    I wasn't suggesting a split. Just noting I wasn't sure there was an existing standout library.

  605. pep.

    Not that it's really usable yet but we'd like to get funding for it so we can spend more time

  606. kokonoe has left

  607. sonny has joined

  608. Shell has left

  609. kokonoe has joined

  610. !XSF_Martin has left

  611. alameyo has left

  612. alameyo has joined

  613. stpeter has joined

  614. !XSF_Martin has joined

  615. !XSF_Martin has left

  616. !XSF_Martin has joined

  617. beta has left

  618. Daniel has left

  619. Daniel has joined

  620. pdurbin has left

  621. stpeter has left

  622. APach has left

  623. APach has joined

  624. lovetox has joined

  625. holger has joined

  626. holger has left

  627. holger has joined

  628. holger has left

  629. holger has joined

  630. marc_ has left

  631. !XSF_Martin has left

  632. !XSF_Martin has joined

  633. marc_ has joined

  634. !XSF_Martin has left

  635. adiaholic has left

  636. adiaholic has joined

  637. waqas has joined

  638. holger has left

  639. holger has joined

  640. !XSF_Martin has joined

  641. holger has left

  642. holger has joined

  643. dwd

    "I’m not immediately sure that what I was saying didn’t make sense." - thanks, Kev, for making that clear...

  644. marc_ has left

  645. holger has left

  646. moparisthebest has joined

  647. mathijs has left

  648. mathijs has joined

  649. holger has joined

  650. Guus

    You mean: "thanks, Kev, for not making that unclear"

  651. holger has left

  652. holger has joined

  653. kokonoe has left

  654. mathijs has left

  655. mathijs has joined

  656. holger has left

  657. holger has joined

  658. holger has left

  659. holger has joined

  660. holger has left

  661. holger has joined

  662. holger has left

  663. holger has joined

  664. UṣL has left

  665. holger has left

  666. holger has joined

  667. stpeter has joined

  668. kokonoe has joined

  669. alameyo has left

  670. alameyo has joined

  671. !XSF_Martin has left

  672. holger has left

  673. holger has joined

  674. holger has left

  675. holger has joined

  676. dwd

    Guus, It'd be wrong to say I'm not certain that wouldn't have been worse.

  677. Kev has joined

  678. Guus

    I agree.

  679. Kev

    You're welcome.

  680. Guus

    (I just guessed a response)

  681. jonas’

    Failed to parse sentence: resource limit exceeded while processing negations.

  682. !XSF_Martin has joined

  683. Yagiza has left

  684. Yagiza has joined

  685. stpeter has left

  686. Kev

    Everyone does like to pick on my English :p

  687. Guus

    In fairness, your Hungarian is worse.

  688. Guus

    (again, guessing)

  689. Kev

    It's not impossible.

  690. dwd

    Guus, My Hungarian used to be pretty good, but these days I just use simpler vairable names and a better IDE.

  691. marc_ has joined

  692. Kev

    dwd: I think you're actually saying in that thread that the IETF now has 4 stages, which is roughly what I'm proposing for the XSF (modelled on Flo's suggestion).

  693. Kev

    That is, I'm suggesting we have Individual I-Ds, which are numberless.

  694. dwd

    Kev, No, the IETF dropped a stage. It did have 4 indeed.

  695. Kev

    And the acceptance for /those/ could be mechanical.

  696. Kev

    It had five, now it has four, no? If you count Individual and WG as distinct.

  697. Daniel

    > That is, I'm suggesting we have Individual I-Ds, which are numberless. I was going to suggest that after what dwd wrote

  698. dwd

    Yes, that's fair.

  699. Kev

    So we could have an entirely mechanical numberless stage that gets it published and under IPR.

  700. Kev

    More or less inbox-plus.

  701. dwd

    FWIW, I'd be fine with having those numbered. Although given that people get excited that we have lots of XEPs, there's that risk.

  702. Kev

    Experimental becomes roughly what it is (effectively) now, which is that it's a direction Council wants to see investigated.

  703. Daniel

    Versioned inbox

  704. Kev

    dwd: I think the numbers are part of the problem.

  705. Kev

    dwd: I think "It's been published and assigned a number" is what makes the issue with accepting knowing it needs significant changes.

  706. pep.

    I really need to catchup on standards@

  707. Kev

    By having the first stage be "It doesn't have a number yet", it makes the risk of it being published and then people saying "Well, it's been standardised, it can't be changed" much lower, I think.

  708. Kev

    1) Published without number. Anything goes, mechanical acceptance (maybe with a check that it's not just "Screw the Establishment" 5000 times or something, but anything goes. 2) Experimental. Anything goes quality-wise, but confirmation it's a direction the Council think should be investigated. 3) Draft as-is (but maybe a lower barrier) 4) Final as-is (but...)

  709. Kev

    Which is actually not a million miles away from how the IETF works, by my understanding. Not strictly equivalent, but it has similar properties.

  710. pdurbin has joined

  711. Kev

    And yes, Daniel, it is very similar to the inbox, in as much as there's no comment on quality, just that it's for future consideration.

  712. Daniel

    > "Well, it's been standardised, it can't be changed" much lower, I think. I think the argument usually goes: we had to implemt something that wasn't ready and ship it because the users were asking for this feature and we couldn't wait five years and now we kinda have to consider dealing with legacy

  713. Daniel

    I think those considerations are independent of numbers or stage names

  714. Kev

    Which I do think is important. I don't think another stage of Council review and etc. is going to add anything.

  715. Kev

    Daniel: I boldy assert that I've heard both types of claims.

  716. Kev

    Nothing we do about versioning or numbering or stages will effect changes in "I had to deploy something now, so I did a thing". But the other side we can do something about.

  717. marc_ has left

  718. Zash

    Hiw do you get the process to handle that?

  719. pdurbin has left

  720. Kev

    Zash: I think the first stage helps with that. You can get it published but not advanced to Experimental, and thus not yet have a number assigned, and still be called Proto-XEP.

  721. dwd

    So what does "Experimental" mean, and what would be the guidance to Council?

  722. Zash

    Won't stop people from deploying and causing the same problem?

  723. Kev

    "Council considers this is an appropriate direction to be directing investigation" or something.

  724. Kev

    So when Council sees a spec and says "This is a problem that needs solving, but this isn't the appropriate way to do it", they can hold it in (1) for a while. It's still been published, it's still versioned, it's still IPRd.

  725. Zash

    I-Ds get deployed and entrenched if people need and implement them

  726. holger has left

  727. holger has joined

  728. Kev

    Zash: Yes, as I said, you can't effect any change in "I had to do a thing so I did a thing".

  729. Kev

    Zash: You can address "This is now a standard, everyone should be implementing it", I hope.

  730. Kev

    So Reactions would have been able to be published by now, even though it needs to be framed in terms of a common approach (e.g. Fastening), it just wouldn't be Experimental yet.

  731. Kev

    Fastening might or might not have reached Experimental (I think the direction is right, FWIW, but that might not be clear enough, who knows).

  732. Kev

    Several of the things (IM2?) I've pushed out full of TODOs because we needed something published for discussion etc. would have stayed out of Experimental, at least for a while.

  733. !XSF_Martin has left

  734. Kev

    I'm not suggesting, FWIW, raising the barrier of Experimental to "This must be implementable and interoperable from the spec as-written". I think it's a good aim for all specs, but I think it's probably fine to leave some hand-waving in places.

  735. MattJ

    I'm in favour of anything that prevents XEPs being assigned numbers while having entire sections consist of "TODO: Figure this thing out"

  736. MattJ

    and then sitting there for years

  737. Daniel

    > I'm in favour of anything that prevents XEPs being assigned numbers while having entire sections consist of "TODO: Figure this thing out" Yes. People at the very least need to be able to write prototype implementations

  738. MattJ

    Interoperable ones, at that

  739. Kev

    As above, I'm not convinced that Experimental needs to be entirely baked to the point that interop is guaranteed. I think it's fine for it to be genuinely an Experiment with questions that still need answering.

  740. Kev

    I'd like to introduce a new stage at the low end, not to life the existing stages.

  741. !XSF_Martin has joined

  742. Daniel

    > I'd like to introduce a new stage at the low end, not to life the existing stages. We can do both. Introduce something at the low end end slightly raise the barrier for experimental (from what is described in xep1) to something which we are kinda doing now already

  743. Daniel

    I think introducing super-inbox also lowers the pressure of trying to become experimental

  744. Kev

    I think that last point is very desirable.

  745. kokonoe has left

  746. Kev

    I would like to still see Experimental able to hold experiments that don't have all the answers. I think requiring it to be fully baked before Experimental wouldn't be the best choice. But I think allowing *everything* relevant to be published seems very desirable.

  747. Kev

    There is then no inbox, essentially. ProtoXEP is a stage in the process, rather than a colloquialism.

  748. Kev

    We might still choose to have them stored in the inbox/ path on the site, for various reasons, but it's a Real Thing rather than just somewhere for Council to look at things on their way to Experimental.

  749. Kev

    (And we could do all this without even formally changing our process, although I suggest it would be a good idea to do so)

  750. pep.

    Daniel, I'm going to be the one picking on details as usual but "stable it has to be stable." doesn't mean much. The XSF would also need to define their version of "stable".

  751. pep.

    (reading the thread)

  752. Zash

    Things might change anyways

  753. pep.


  754. SpaceFreak aka Tracer has left

  755. Daniel

    Ultimately the decision on how much leftover TODOs are OK will still be acceptable. But if experimental becomes (as it de-facto has been for a while now) a 'only thing that council agrees is a good direction' and not 'everything goes' as xep1 spells it out to be I think I want more formal readiness (meaning not _just_ TODOs)

  756. pep.

    To me this thread and discussions scream "People are afraid of change"

  757. pep.

    Introducing or removing a stage won't change this I believe

  758. Daniel

    Otherwise it sends a weird message when xeps that "some random people on the internet don't agree with" become accepted and on the other hand we have xeps that are really just stubs someone wrote in 20 minutes

  759. pep.

    Final is never final. Another XEP someday will come to replace it

  760. pep.

    (And to me it's exactly the same as changing that Final XEP in the first place..)

  761. Daniel

    Otherwise it sends a weird message when xeps that "some random people on the internet don't agree with" don't get accepted and on the other hand we have xeps that are really just stubs someone wrote in 20 minutes

  762. Kev

    Daniel: To be clear, I'm not suggesting that Experimental should be a stub. Just that it doesn't have to be complete. I have certainly published XEPs to Experimental with my name on that I would not have needed to be Experimental if we had the ProtoXEP stage.

  763. stpeter has joined

  764. Kev

    I'm also fine for Experimental to remain a judgement call of Council.

  765. Kev

    And I'm also fine with trying to update XEP1 to describe Experimental as it is being used currently, rather than necessarily what it suggests, if it suggests otherwise.

  766. Kev

    And then Peter appears for an informed opinion :)

  767. marc_ has joined

  768. intosi has left

  769. marc_ has left

  770. calvin has joined

  771. mtavares has left

  772. stpeter has left

  773. patrick has joined

  774. mtavares has joined

  775. holger has left

  776. marc_ has joined

  777. kokonoe has joined

  778. intosi has joined

  779. mukt2 has joined

  780. karoshi has left

  781. karoshi has joined

  782. Steve Kille has joined

  783. Wojtek has left

  784. karoshi has left

  785. karoshi has joined

  786. j.r has left

  787. Maranda has left

  788. Maranda has joined

  789. Wojtek has joined

  790. mathijs has left

  791. marc_ has left

  792. calvin has left

  793. calvin has joined

  794. Steve Kille has left

  795. mathijs has joined

  796. mukt2 has left

  797. mukt2 has joined

  798. Steve Kille has joined

  799. mathijs has left

  800. !XSF_Martin has left

  801. !XSF_Martin has joined

  802. mukt2 has left

  803. mathijs has joined

  804. mathijs has left

  805. pdurbin has joined

  806. mathijs has joined

  807. kokonoe has left

  808. mukt2 has joined

  809. j.r has joined

  810. pdurbin has left

  811. !XSF_Martin has left

  812. marc_ has joined

  813. stpeter has joined

  814. larma has left

  815. !XSF_Martin has joined

  816. larma has joined

  817. calvin has left

  818. marc_ has left

  819. kokonoe has joined

  820. mukt2 has left

  821. patrick has left

  822. kokonoe has left

  823. marc_ has joined

  824. stpeter has left

  825. mathijs has left

  826. mathijs has joined

  827. kokonoe has joined

  828. marc_ has left

  829. !XSF_Martin has left

  830. mukt2 has joined

  831. intosi has left

  832. !XSF_Martin has joined

  833. mathijs has left

  834. mathijs has joined

  835. Arc

    pep.: I looked at xmpp-rs but is it asyncio?

  836. pep.

    "which one", atm it's using tokio behind, we started having a look at async-std. tokio 0.2 uses the keywords fwiw

  837. Arc

    yea I was looking at the tokio one

  838. mukt2 has left

  839. Arc

    this is for playerd and therefore two basic requirements; async and good pubsub support

  840. pep.

    What's playerd?

  841. mukt2 has joined

  842. pep.

    "good pubsub support" is not there at all. (We barely have messages in that high-level API), but I'd happily work on that :)

  843. mathijs has left

  844. mathijs has joined

  845. Arc

    in short playerd is intended to provide game console '-like player management cross-platform; ie, input->player mapping, user profiles, and the basic XMPP-IM functionality like friends lists and chat

  846. Arc

    imagine on any game console what happens when you press the system button on a controller

  847. pep.

    I see

  848. Arc

    most games don't support this on any non-console platform. even on Steam (which has the best of this right now) when you load a game with two players you end up having to physically swap controllers when the game opens and you discover you have them backwards

  849. mukt2 has left

  850. gav has left

  851. gav has joined

  852. mukt2 has joined

  853. !XSF_Martin has left

  854. marc_ has joined

  855. Arc

    btw we have a whole new group of teenagers discovering the joys of XMPP right now

  856. pep.


  857. Arc

    copyleft games

  858. Arc

    gci@chat.copyleftgames.org MUC

  859. dwd

    Arc, This library able to be used in commercial games, or only GPL?

  860. pep.

    Pidgin is so much fail :x

  861. Arc

    dwd: playerd is intended to be permissive

  862. !XSF_Martin has joined

  863. Arc

    the code itself is AGPL right now though. the intention is to put a LGPL-like exception on it to allow it to be used by non-copyleft games

  864. dwd

    Arc, Cool. I've heard some rumblings that Epic and others are struggling with XMPP, but perhaps some demonstrations that it has broader utility might help.

  865. Arc

    valve reached out to us about playerd a bit ago too for the same reason, which gave me some hope that steam could broaden their xmpp support

  866. dwd

    Arc, Does Steam use XMPP at all?

  867. pep.

    That's what I understand from this sentence :x

  868. Arc

    from what little I could gleam (since the valve person was under NDA that I didn't sign) they have XMPP at some place internally but currently expose some "Steamworks IM" protocol to games, and there's a lot of pressure from gamedevs to use XMPP instead, so that led them to look into offering an alternative protocol

  869. dwd

    Oh, interesting. Perhaps Board could try to get into contact and see what we could offer?

  870. marc_ has left

  871. stpeter has joined

  872. pep.

    What can the XSF offer at all? "Hey we can propose you more protocols!" :)

  873. dwd

    I don't know, but some contact might answer that question.

  874. mukt2 has left

  875. Arc

    they dont need more protocols :-)

  876. Arc

    playerd uses common sense stuff

  877. Zash

    which XEP is that;

  878. mukt2 has joined

  879. Wojciech Kapcia has joined

  880. Wojciech Kapcia has left

  881. Nekit has left

  882. kokonoe has left

  883. mukt2 has left

  884. kokonoe has joined

  885. arc has left

  886. arc has joined

  887. mukt2 has joined

  888. sjaak has left

  889. sjaak has joined

  890. Yagiza has left

  891. pdurbin has joined

  892. Kev has left

  893. !XSF_Martin has left

  894. Wojtek has left

  895. Wojtek has joined

  896. pdurbin has left

  897. mukt2 has left

  898. mukt2 has joined

  899. !XSF_Martin has joined

  900. holger has joined

  901. calvin has joined

  902. arc has left

  903. arc has joined

  904. Wojtek has left

  905. sjaak has left

  906. sjaak has joined

  907. sjaak has left

  908. sjaak has joined

  909. sjaak has left

  910. sjaak has joined

  911. sjaak has left

  912. sjaak has joined

  913. sjaak has left

  914. sjaak has joined

  915. sjaak has left

  916. sjaak has joined

  917. kokonoe has left

  918. sjaak has left

  919. sjaak has joined

  920. kokonoe has joined

  921. debacle has left

  922. Nekit has joined

  923. !XSF_Martin has left

  924. Arc

    for playerd? I haven't submitted one

  925. adiaholic has left

  926. Arc

    btw do we have a XEP for Oauth2 SASL negotiation?

  927. calvin has left

  928. calvin has joined

  929. Arc

    as in RFC 7628 as applied to XMPP? even a best practices one?

  930. !XSF_Martin has joined

  931. APach has left

  932. Arc

    XMPP is referred to in the RFC

  933. sjaak has left

  934. sjaak has joined

  935. Arc

    but the RFC more or less starts after you have the bearer token. it doesn't really dig deep into how XMPP clients get that token

  936. Zash

    I also seem to miss how that's supposed to work.

  937. Arc

    oauth2 is a pretty big spec and covers use cases like keyboard-less UI.

  938. Arc

    consider a teen with the tech savy of the average iphone user, sitting down with a game controller and wants to "log in. oauth2 supports this, but its not very obvious how the xmpp client would.

  939. Zash

    I kinda get the impression that OAuth only works if you're web-based.

  940. Arc

    ideally they'll get a short URL plus a qr code on the screen to access on their phone to log in via phone-based web browser, or any other browser

  941. Arc

    nope. it works in all cases - and that's my point

  942. Arc

    you can log your smart fridge in with no form of input at all

  943. beta has joined

  944. Arc

    fridge would just give you a URL to access to log it in

  945. Arc

    like, "Go to HTTPS://AUTH.GE.COM/H8C0 to login" with an all-caps qrcode (all caps qrcodes use fewer bits to encode so easier bigger squares, easier to scan)

  946. beta has left

  947. stpeter has left

  948. beta has joined

  949. Arc

    the fridge keeps accessing a https url and receives a authorization_pending error response over and over until you actually log it in. and if you take very long the server will send a "expired_token" error, in which case the fridge will have to start over often with a new URL.

  950. Arc

    but once you login on your phone or laptop or whatever, the fridge will get a response containing the bearer token that it can use to actually login

  951. sjaak has left

  952. sjaak has joined

  953. Arc

    usually the server will wait a few seconds on each request before sending the authorization_pending error, so that the exact moment the login is completed the fridge can be sent the bearer token. in production there's virtually no delay at all.

  954. Arc

    desktop XMPP clients can do the same; you just launch a web browser on the url instead of showing it to them on a screen. you don't need a response from the browser.

  955. matkor has left

  956. matkor has joined

  957. Zash

    "just launch a web browser" :(

  958. Arc

    desktop client runs "browser-open" command with url "https://login.google.com/bla". it doesn't need to get anything back from the browser

  959. Arc

    there's a standard way to open a URL via default browser on every modern OS

  960. jonas’

    so, yes, web-only

  961. Arc

    web-only for login yes. but not for web XMPP clients only

  962. j.r has left

  963. Arc

    here's two lines of Python code to do this from every OS: import webbrowser webbrowser.open(oauth2_url)

  964. andrey.g has left

  965. Arc

    it will return True.

  966. Arc

    seriously the whole client handshake is maybe 30 lines of python, tops. my point being is if you're not versed in oauth2 this is very non-obvious so a XEP would be useful. IMHO every XMPP client should be using bearer tokens for account security. for account security every XMPP server should only accept oauth2 bearer tokens.

  967. holger has left

  968. Zash

    But, requiring a web browser, I don't want that

  969. Zash

    Assume that I explode if I have to evaluate JavaScript. How do I OAuth?

  970. j.r has joined

  971. krauq has left

  972. Guus

    Preferably at a safe distance from other combustibles.

  973. mathijs has left

  974. mathijs has joined

  975. pdurbin has joined

  976. sjaak has left

  977. sjaak has joined

  978. mathijs has left

  979. mathijs has joined

  980. calvin has left

  981. sonny has left

  982. pdurbin has left

  983. j.r has left

  984. sjaak has left

  985. sjaak has joined

  986. mr.fister has joined

  987. kokonoe has left

  988. Steve Kille has left

  989. !XSF_Martin has left

  990. larma

    Zash, I don't think there is a reason why OAuth would require JavaScript (beside most implementations certainly will)

  991. krauq has joined

  992. !XSF_Martin has joined

  993. Arc

    Zash: OAuth is a web api. but its completely conceivable, if you write your own oauth2 web client, to use a simple HTTPS form. oauth2 can be implemented entirely server-side

  994. Arc

    or better yet write it in rust and compile to wasm

  995. kokonoe has joined

  996. Arc

    without a web browser at all is hard since oauth2 is a standardized web api

  997. calvin has joined

  998. david has left

  999. david has joined

  1000. adiaholic has joined

  1001. stpeter has joined

  1002. j.r has joined

  1003. calvin has left

  1004. Nekit has left

  1005. !XSF_Martin has left

  1006. Ge0rG

    I didn't know that browsers are the only operating system for APIs

  1007. !XSF_Martin has joined

  1008. adiaholic has left

  1009. sjaak has left

  1010. sjaak has joined

  1011. sjaak has left

  1012. sjaak has joined

  1013. calvin has joined

  1014. calvin has left

  1015. calvin has joined

  1016. !XSF_Martin has left

  1017. !XSF_Martin has joined

  1018. calvin has left

  1019. calvin has joined

  1020. calvin has left

  1021. debacle has joined

  1022. Arc

    Ge0rG: lets be real, in the real world 95%+ of software devs are web developers and we're <<looks at the XSF membership>> far too few to write everything. But web devs are limited, and thankfully still need us or we'd all be unemployable.

  1023. sjaak has left

  1024. sjaak has joined

  1025. sjaak has left

  1026. sjaak has joined

  1027. kokonoe has left

  1028. Arc

    the only reason these companies approach us is because some web developer sent them our way. in literally every case.

  1029. Arc

    when I get an email from taiwan about working on a smart appliance its because their web devs have already tried and failed. they're hiring a "specialist" because that's what we've all become. and they've already developed the web-side

  1030. Arc

    the most extreme case you'll get consensus from their inhouse webdevs is to use an existing web standard

  1031. Arc

    half the products I get to work on are running NodeMCU (ESP8266) and they already tried and failed to run the whole appliance on javascript using web apis.

  1032. Arc

    ... and that's why a vast majority of IoT is running stupid JSON-based web apis

  1033. Arc

    and it doesn't even matter because its for a single production run. the company exists only that long. they sell 100k of a product with a $40 margin over cost to manufacture, the 4 million pays the employees and investors, and poof they're gone with no continued support.

  1034. karoshi has left

  1035. Arc

    the developers working for these companies have a strictly disposible attitude.

  1036. Arc

    they don't care about standards. the only reason they use standards is to shorten their development cycle. they don't care about license compliance. if we're super lucky they toss their code over the wall on github to bitrot.

  1037. Lance has joined

  1038. pep.

    Great pitch for all these new students that want to get into programming.

  1039. pep.

    I would have gone farming right away if I had known

  1040. Arc

    lol well :-)

  1041. !XSF_Martin has left

  1042. pdurbin has joined

  1043. kokonoe has joined

  1044. mukt2 has left

  1045. mukt2 has joined

  1046. Arc

    my point is standards are only useful if people use them, we need to make XMPP accessible to web devs because they control the code that goes into products, and they also control who gets funded.

  1047. !XSF_Martin has joined

  1048. intosi has joined

  1049. pdurbin has left

  1050. sjaak has left

  1051. sjaak has joined

  1052. paul has left

  1053. sjaak has left

  1054. sjaak has joined

  1055. intosi has left

  1056. marc_ has joined

  1057. sjaak has left

  1058. sjaak has joined

  1059. sjaak has left

  1060. sjaak has joined

  1061. goffi has left

  1062. !XSF_Martin has left

  1063. !XSF_Martin has joined

  1064. stpeter has left

  1065. mukt2 has left

  1066. mukt2 has joined

  1067. !XSF_Martin has left

  1068. !XSF_Martin has joined

  1069. sjaak has left

  1070. sjaak has joined

  1071. sjaak has left

  1072. sjaak has joined

  1073. mukt2 has left

  1074. mr.fister has left