XSF Discussion - 2019-08-22

  1. stpeter has left
  2. Fahrgast has left
  3. Daniel has left
  4. Daniel has joined
  5. Fahrgast has joined
  6. zach has left
  7. zach has joined
  8. jmpman has left
  9. Daniel has left
  10. Fahrgast has left
  11. Allo has left
  12. Fahrgast has joined
  13. Daniel has joined
  14. Allo has joined
  15. Daniel has left
  16. Daniel has joined
  17. UsL has left
  18. UsL has joined
  19. aj has joined
  20. stpeter has joined
  21. peter has joined
  22. adityaborikar has joined
  23. aj has left
  24. patrick has left
  25. zach has left
  26. zach has joined
  27. Chobbes has left
  28. adityaborikar has left
  29. adityaborikar has joined
  30. pdurbin has joined
  31. Daniel has left
  32. adityaborikar has left
  33. marc_ has left
  34. Daniel has joined
  35. pdurbin has left
  36. marc_ has joined
  37. neshtaxmpp has left
  38. neshtaxmpp has joined
  39. karoshi has joined
  40. peter has left
  41. zach has left
  42. zach has joined
  43. pdurbin has joined
  44. stpeter has left
  45. adityaborikar has joined
  46. Daniel has left
  47. Daniel has joined
  48. adityaborikar has left
  49. jmpman has joined
  50. zach has left
  51. zach has joined
  52. zach has left
  53. zach has joined
  54. Daniel has left
  55. Fahrgast has left
  56. Fahrgast has joined
  57. Daniel has joined
  58. zach has left
  59. zach has joined
  60. Lance has joined
  61. adityaborikar has joined
  62. jmpman has left
  63. tom has joined
  64. marc_ has left
  65. jmpman has joined
  66. andy has joined
  67. adityaborikar has left
  68. marc_ has joined
  69. Yagiza has joined
  70. adityaborikar has joined
  71. waqas has left
  72. zach has left
  73. zach has joined
  74. Daniel has left
  75. Daniel has joined
  76. Fahrgast has left
  77. zach has left
  78. zach has joined
  79. Lance has left
  80. pdurbin has left
  81. Tobias has joined
  82. pdurbin has joined
  83. Fahrgast has joined
  84. jmpman has left
  85. Lance has joined
  86. Daniel has left
  87. Daniel has joined
  88. Nekit has joined
  89. aj has joined
  90. Daniel has left
  91. wurstsalat has joined
  92. Daniel has joined
  93. Lance has left
  94. zach has left
  95. zach has joined
  96. jabberjocke has left
  97. goffi has joined
  98. Lance has joined
  99. murabito has left
  100. murabito has joined
  101. jonas’ what has time to do with that?
  102. adityaborikar has left
  103. zach has left
  104. zach has joined
  105. Daniel Yes you need to start mam or get the last if *before* you send presence
  106. Daniel Yes you need to start mam or get the last ID *before* you send presence
  107. Daniel Don't think carbons have anything to do with that
  108. jonas’ you could enable carbons before sending presence, too
  109. Tobias has left
  110. Tobias has joined
  111. jonas’ but don’t you need to to MAM *after* sending presence/enabling carbons to not lose a message?
  112. jonas’ otherwise, if you got the last ID / queried MAM *before* sending presence, a message might’ve arrived between querying the ID and sending presence/enabling carbons, which you’d then miss
  113. Daniel Mhhh ideally then you'd get the last ID before presence but fire the query after
  114. jonas’ sure
  115. jonas’ no, wit
  116. jonas’ *wait
  117. jonas’ if you get the last ID before presence, what happens if a message arrives between you getting the last ID and you sending presence?
  118. Daniel jonas’: last ID defines the lower end
  119. jonas’ what does "lower" mean in that context?
  120. Daniel Upper end is undefined
  121. jonas’ is last ID the last thing you saw in your last session?
  122. jonas’ or is last ID the newest ID?
  123. Daniel Yes
  124. Daniel The newest you currently have locally
  125. ralphm But you won't get newer messages until you sent presence.
  126. jonas’ Daniel, I see!
  127. jonas’ that makes sense then
  128. jonas’ it’s early in the morning, sorry :)
  129. sezuan has joined
  130. ralphm Daniel: if you're saying there's a specific order of events that ensures you get all messages, is that documented somewhere?
  131. david has left
  132. ralphm Like: get last id, send carbons request, send presence, request mam?
  133. ralphm Because XEP-0313 says client sync is it of scope (!), and refers to bind2.
  134. ralphm out of scope
  135. Ge0rG ralphm: an attempt was made to document it in 0313, but it turned out not to work.
  136. Daniel what didn’t work?
  137. Daniel the attempt?
  138. Ge0rG Daniel: the actual ordering that was written down, I think
  139. ralphm Ge0rG: so is there an order that does work? Or can it not be solved with current protocols?
  140. Daniel the order that ralphm just wrote down doesn’t work?
  141. Ge0rG So you note the last id you know, send presence, and then fetch MAM in parallel to receiving offline messages / live traffic? Is there any change to sync linearly in the actual message order?
  142. ralphm I believe that the order I mentioned is at-least-once. There's no such thing as exactly-once in general.
  143. Daniel well receiving MAM and offline messages in parallel isn’t that bad; you can dedup them
  144. Holger Right, there's no way to avoid dedup.
  145. Daniel however when i detect that MAM preferences are set to always and server has offline msg mangament things i also run a purge
  146. Daniel before sending presence
  147. Holger … which can reduce the number of messages to dedup, yes.
  148. Daniel but i see that only as a minor traffic optimization
  149. Ge0rG You also need to dedup between the Carbons of live messages that arrived between you enabling Carbons and you sending presence, and their respective offline copy
  150. ralphm Daniel: is the filter for messages going to MAM and offline identical?
  151. Ge0rG ralphm: probably not
  152. Daniel my dedup would work fine; and does on servers that don’t have the purge thing
  153. Daniel ralphm, i guess there is nothing in the protocol to ensure that
  154. ralphm Ge0rG: you don't get carbons until you send presence
  155. Ge0rG Daniel: if you purge offline, isn't there a chance that you'll purge too much?
  156. Ge0rG ralphm: are you sure?
  157. Daniel Ge0rG, well my reasoning is if it doesn’t end up in MAM it wasn’t important
  158. Daniel your second client wouldn’t receive it either
  159. ralphm That's the idea. Before sending presence, you only get message stanzas directed at your full JID.
  160. Daniel i mean if the rules between offline messages and mam differ it is probably time to reconsider the rules
  161. Ge0rG ralphm: I don't see presence mentioned in 0280: > When the server receives a <message/> eligible for carbons delivery addressed to a client JID (either bare or full), it delivers the <message/> according to RFC 6121 § 8.5.3, and then delivers a forwarded copy to each Carbons-enabled resource for the matching bare JID recipient that did not receive it under the RFC 6121 delivery rules.
  162. Daniel but if you hate the purge don’t purge
  163. Daniel that's not something i would codify anywhere
  164. Daniel especially since purge capability isn’t really widespread anywhere
  165. ralphm Ge0rG: hmm, that surprises me. Curious how servers implement this then.
  166. Steve Kille has left
  167. zach has left
  168. zach has joined
  169. jonas’ I’m pretty sure that some people use carbons-enabled clients with negative priority to get all messages (also) on a non-carbons-enabled client :)
  170. Ge0rG Daniel: I remember complaining about a race condition there, but I can't say right now what it was
  171. Ge0rG jonas’: I'm doing that, but a negative presence still is a presence
  172. ralphm Right
  173. ralphm Just for my information, though, the only upshot of that combination (negative presence and carbons) is that you don't get message stanzas to bare JID that don't match the carbon criteria?
  174. ralphm I'm curious in what situation that is beneficial.
  175. jonas’ ralphm, the non-carbons-enabled client will get all the messages because it’s the only one with >= 0 priority
  176. jonas’ while the carbons-enabled clients also get the messages because they’ve enabled carbons
  177. ralphm But only of type chat, for example.
  178. Holger So just a trick to get carbons behavior with n clients although only n-1 clients support carbons.
  179. sezuan has left
  180. lovetox_ has joined
  181. jonas’ "good enough"
  182. ralphm Hmm
  183. Daniel not that implementing carbons in a client is hard…
  184. ralphm It is interesting though. Most clients use presence priority to figure out which 'show' to display.
  185. ralphm So in this setting that might mislead
  186. ralphm Obscure edge case, maybe
  187. Holger Yes.
  188. sezuan has joined
  189. ralphm Any way, I think that servers will still wait for presence to start sending any messages.
  190. ralphm (that were not to this connection's resource)
  191. Lance has left
  192. waqas has joined
  193. Steve Kille has joined
  194. lovetox_ It does not matter in what order you activate carbons, presence, offline messages
  195. lovetox_ you can never miss a message if you do MAM last
  196. lovetox_ thats the easy way, that might have here and there some message that has to be deduped
  197. lovetox_ but thats no problem with stanza.-id
  198. jubalh has joined
  199. ralphm lovetox_: what was said is that before sending all that, you need to establish the last id you have.
  200. murabito has left
  201. murabito has joined
  202. ralphm The rest of the order is less relevant, although there might be an optimum to minimize traffic of dupes.
  203. jcbrand has joined
  204. lovetox_ you have the last ID in your database
  205. zach has left
  206. zach has joined
  207. ralphm Yes, you just need to make sure you don't invalidate it by getting new messages before doing MAM
  208. lovetox_ yes of course, before MAM catchup is not complete you should not record "last stanza id"
  209. lovetox_ and MAM catchup will always give you the very last message
  210. ralphm That was the thing we tried to convey
  211. ralphm I'm still curious as to why it is said to be out of scope and mentions bind2 as possible remedy.
  212. flow So is it guaranteed that servers don't mix in live messages while sending MAM messages?
  213. ralphm flow: I don't think so
  214. ralphm That seems complex to implement
  215. Lance has joined
  216. jonas’ ralphm, I think the idea was to have a single sync point where you get the ID of the most recent message in your MAM (or possible multiple IDs based on the sender JID) and enable carbons and message delivery in a single server-atomic operation
  217. jonas’ removing the need to deduplicate there
  218. waqas has left
  219. Ge0rG Yes, essentially you need an atomic operation where you enable live traffic and receive the ID of the last non live message
  220. Ge0rG So that you can fill the hole.
  221. Ge0rG But there is no way to actually receive what you need in a linear order.
  222. jonas’ true
  223. jmpman has joined
  224. david has joined
  225. david has left
  226. Ge0rG and there is no way to receive the messages exactly once.
  227. lovetox_ flow why would it make a difference if a server mixes in live messages?
  228. jonas’ Ge0rG, with bind2, there would be, or?
  229. Ge0rG jonas’: maybe
  230. jonas’ "maybe"?
  231. Ge0rG jonas’: if bind2 also removes all MAMed messages from the offline history sync
  232. jonas’ in my world, offline messages do not exist and you always purge them.
  233. david has joined
  234. ralphm jonas’: that sync point seems to be when the server receives presence
  235. jonas’ ralphm, it’s not a sync point, because you don’t get the "newest MAM ID" from that
  236. jonas’ (you = the client)
  237. ralphm It is a sync point for live messages
  238. jonas’ how is it sync if it’s one-way?
  239. ralphm Well, sure you still need to do the mam query afterwards. I don't see how one can technically achieve exactly once.
  240. jonas’ ralphm, if the server replied (to the presence, or to whatever) with the ID of the most-recent stanza in the MAM archive at the time the presence was received
  241. jonas’ then you’d query from "last stanza ID I’ve seen" until "most-recent stanza ID I got from presence"
  242. lovetox_ you dont record that jonas
  243. zach has left
  244. zach has joined
  245. jonas’ lovetox_, ?
  246. lovetox_ you dont record last-stanza-id until you made your mam query
  247. lovetox_ and while the mam query is running you ignore all stanza-ids from live and carbons
  248. ralphm From a client perspective they'd be ideal. I don't think that's easy to implement server side. Also, exactly once delivery is theoretically impossible.
  249. lovetox_ you only look at the mam query
  250. jonas’ ralphm, sure.
  251. lovetox_ the last message in the mam query, is your most recent message, then you record the stanza-id
  252. lovetox_ and then you are fully synced
  253. lovetox_ and it does not matter what the server sends you inbetween your mam query
  254. jonas’ if you don’t want to be efficient, sure.
  255. ralphm I've also seen MAM implementations that were eventually consistent, by the way.
  256. lovetox_ yes jonas thats what we were saying
  257. ralphm (server side)
  258. lovetox_ and nobody needs to be efficient here, mam is complex enough
  259. lovetox_ a few duplicates are not bad, its totally fine
  260. lovetox_ and you can now overengineer that so there will never be a message twice, but its not an "issue" of clients that we are so inefficient because we get 3 offline messages while mam query
  261. lovetox_ and btw you have the same thing with MUC
  262. lovetox_ live messages will arrive while you query mam
  263. ralphm Oh, don't get me started on MUC or MIX.
  264. lovetox_ i would argue this was a problem back in mam:0 mam:1 days
  265. lovetox_ if you could not depend on stanza-id
  266. lovetox_ now with mam:2, really i dont care about a handful of duplicates in a mam catchup
  267. ralphm Sure.
  268. ralphm Problem with group chats is that you have to do it for each room.
  269. ralphm And although MIX currently says you should also store in your local archive (which I like, as it decreases client complexity), it is unclear how a server catches up to the room archive in case of delivery issues (e.g. s2s outages).
  270. Mikaela has joined
  271. Nekit has left
  272. morgan has joined
  273. Mikaela has left
  274. lovetox_ ralphm what is the problem with making a query to each MUC? inefficient?
  275. Mikaela has joined
  276. lovetox_ yes maybe i can spare 20 stanzas, for 20 muc joins, but other than that, this just works
  277. lovetox_ if i switch now to some other concept, like you just decribe and for that get real problems like s2s catchup etc
  278. lovetox_ i would stay with the old behaviour
  279. murabito has left
  280. murabito has joined
  281. Daniel Oh I didn't know that the server was supposed to catch up on your behalf. I thought you were just going to miss messages
  282. Daniel Or magically know where the gaps are
  283. morgan has left
  284. remko has joined
  285. Nekit has joined
  286. Ge0rG lovetox_: on mobile clients, it makes sense to reduce: round-trips, wakeups, data bandwidth
  287. lovetox_ yes, i have to trust you on that
  288. lovetox_ though i would say mobile clients with stream management should not run to often into the, total connection reset scenario
  289. zach has left
  290. zach has joined
  291. Daniel Ge0rG, well you'd do mam and join at the same time. so it's not more roundtrips or more wake ups
  292. Daniel all it does it redcue traffic
  293. Daniel which is fair
  294. ralphm Daniel: well, let's say it is unclear. The reason I dislike having to check each channel for its archives, is that it doesn't scale. If you want to implement something like Slack, you can easily be in 40+ channels. In WhatsApp, my daughter is in uncountable ones that are never left.
  295. Ge0rG ralphm: yes, we need a way to start a session by just telling the server "my last local ID is X, give me everything relevant after that"
  296. morgan has joined
  297. pdurbin has left
  298. lovetox_ has left
  299. Daniel ralphm, depends on what you want to scale and how it is implemented server side. send a message to a group of 500 people - you might not want to store it 501 times in different MAM archives
  300. Daniel especially not in 501 transactions
  301. ralphm Daniel: I strongly believe that the collective archive of all my conversations should be as close to me as possible. I.e. my own server.
  302. ralphm I'm sure there are server side optimizations that could be done.
  303. Ge0rG Daniel: having 501 archives wasn't a problem for the designers of MIX
  304. Daniel Ge0rG, yes we are talking about mix
  305. Daniel (or muc/sub which does similar things for that matter)
  306. lovetox_ has joined
  307. ralphm I mean we are also sending 501 copies over the wire.
  308. Nekit has left
  309. Daniel ralphm, i get that. i'm with you on that. however if we don’t change how MAM is currently implemented it will trigger 501 transactions
  310. Ge0rG Daniel: I think the only actual efficiency problem about MIX is that you end up with 501 copies of each message, with 490 accounts being abandoned by their owner
  311. ralphm Even if participants are on the same server. And not counting c2s copies
  312. Daniel which is actually a problem we are running into on one project
  313. moparisthebest has left
  314. Ge0rG Daniel: a server implementing both the MIX and the account domain could use the same archive table, and just remember for each account when it last joined
  315. ralphm If the problem is retention, then there are two ways: limit in time/space, or pay for storage.
  316. Ge0rG speaking of MAM, where are MUC-PMs stored?
  317. Daniel for us it was not necessary the storage but the throughput
  318. Daniel 501 transactions are expensive
  319. Ge0rG is still struggling with the right SQL query to obtain the last-stanza-ID for MAM sync
  320. ralphm Daniel: did you see my 'eventually consistent' remark above?
  321. morgan has left
  322. Daniel ralphm, your point being we just have to do it anyway?
  323. lovetox_ Ge0rG on your account MAM
  324. lovetox_ if this was the question
  325. Ge0rG lovetox_: yes
  326. lovetox_ only type=groupchat is in the MUC archive, usually on current servers
  327. ralphm Daniel: I'd prefer to have one, complete, archive per user, and only use room archives to catch up with pre-join history, but I'm not sure if this is possible. It would surely involve backfilling, and I think that's at the very least undefined, and impossible at worst, with the way MAM is designed right now.
  328. Daniel yes you get your PMs from your normal catchup
  329. morgan has joined
  330. Nekit has joined
  331. Daniel ralphm, yes to all that
  332. Dele (Mobile) has joined
  333. Daniel and also rather expensive with the way MAM is designed right now
  334. Holger Availability of room history is deemed critical enough to dive into the headaches of maintaining local copies even though availibility of the room service itself isn't considered critical enough to distribute it across domains?
  335. Lance has left
  336. Daniel did someone say matrix?
  337. Holger So we get the cons of Matrix without the pros?
  338. Holger Hah.
  339. waqas has joined
  340. Holger Not sure we need to agree on a model though. I guess local MIX archives can remain an optional feature without introducing trouble?
  341. lskdjf has joined
  342. Daniel as long as it is advertised and clients know what to do
  343. morgan has left
  344. lovetox_ Holger just out of interest how many messages does the conversations.im server MAM archive contain currently ?
  345. Daniel alot.jpg
  346. lovetox_ i think you have no MAM storage time limit on that server if i remember correctly
  347. Daniel we do; because we ran out of disk space :-)
  348. lovetox_ haha 😃
  349. larma has joined
  350. Holger This came totally unexpected.
  351. Nekit has left
  352. Nekit has joined
  353. adityaborikar has joined
  354. Dele (Mobile) has left
  355. ralphm Holger: sure, Matrix solves some these issues, but I believe they haven't tackled group scalability fully yet, and if we're talking about possibly having dimensions in MAM archives for reactions, edits, and other attachments to messages, I am not sure it handles that well at all.
  356. zach has left
  357. zach has joined
  358. Yagiza has left
  359. Yagiza has joined
  360. ralphm Having just a dag of commits is not great in that respect
  361. Ge0rG couldn't an account server "cheat" by just forwarding all MAM queries to the MIX service?
  362. jonas’ Ge0rG, only if you were to filter by the MIX service
  363. jonas’ my understanding is that a normal MAM query would give you MIX and non-MIX messages interleaved
  364. ralphm Ge0rG: that might not complete in a reasonable time, and still leave you with gaps?
  365. jmpman has left
  366. ralphm What jonas’ says
  367. Daniel also how would your server know the relevant stanza-id of the mix service
  368. Ge0rG Eww.
  369. Ge0rG Right. So many problems.
  370. ralphm As I said, this is not easy
  371. ralphm By the way, somebody remarked it being ok to send MAM along with join, but join is a one time thing in MIX. After that your server will just send presence.
  372. Ge0rG that remark was about MUC
  373. ralphm Ok
  374. Ge0rG If only we had threads.
  375. morgan has joined
  376. ralphm 😭
  377. waqas has left
  378. adityaborikar has left
  379. jonas’ is this the "Can of Worms" day?
  380. Ge0rG So to sum it up: Carbons and MAM are ugly workarounds to make an unreliable message "delivery" protocol more reliable, and MIX is codifying those workarounds without solving the actual reliability problems?
  381. Ge0rG jonas’: like every day.
  382. Nekit has left
  383. ralphm Ge0rG: maybe.
  384. Lance has joined
  385. Ge0rG So I need to have a flag for type=groupchat messages, to exclude those from the last-id-query, while retaining other-typed messages from MUC JIDs, like invitation and invitation rejection.
  386. Nekit has joined
  387. Daniel yes
  388. Ge0rG And I need a DB index on that flag.
  389. Daniel or do the gajim where you store the last mam ids seperatly
  390. Ge0rG What could possibly go wrong with that?
  391. Daniel probably the same amount of things that can go wrong with your approach
  392. Daniel only different things
  393. Ge0rG I'm still wondering, given a set of IDs, how do I determine which one is last?
  394. Ge0rG The anticipated answer is: I can't, I need to know from the context
  395. zach has left
  396. zach has joined
  397. Daniel the gajim way(tm) also gives you the ability to remove messages locally while still making the catchup work
  398. Daniel and not re-fetch a deleted message
  399. aj has left
  400. Daniel there are forked versions of Conversations that do the gajim for that purpose
  401. morgan has left
  402. Ge0rG Yeah, that makes some sense.
  403. lovetox_ Ge0rG i would suggest to hold the last stanza id per jid, in a separate table
  404. lovetox_ and yes stanza-id has no order
  405. Ge0rG lovetox_: is there anything else one should put into that table?
  406. lovetox_ so just from the id you cant determine if its the highest or lowest
  407. lovetox_ its opaque
  408. Ge0rG lovetox_: you update that table when you finish a MAM query?
  409. lovetox_ no on every MAM message
  410. lovetox_ so if the catchup breaks up or my client crashes
  411. Ge0rG lovetox_: on every in-order MAM message? Or are all your MAM messages in-order?
  412. lovetox_ i dont have to refetch
  413. Ge0rG Sigh. I wish MAM would go away and server would just track the last delivered message per client resource.
  414. Ge0rG it would be so easy with 0198, at least for the sake of full-sync clients
  415. lovetox_ i contact you later Ge0rG with the db schema and what i store, must go now
  416. Ge0rG lovetox_: thanks!
  417. flow lovetox_> flow why would it make a difference if a server mixes in live messages? I guess it depends if a live message triggers something that touches the processing of the incoming MAM archive messages. I was just wondering what people think about that
  418. ralphm For me Gajim never seems to properly retrieve / show MUC history after coming online.
  419. Ge0rG What's the status of the Badges poll, BTW? I'd _really_ like to know the results.
  420. morgan has joined
  421. Daniel has left
  422. Daniel has joined
  423. lovetox_ ralphm this works. but it only shows what you missed
  424. lovetox_ so if you join twice, all history messages are in the history window CTRL + H
  425. Ge0rG what is a history window and why should it even be there?
  426. jmpman has joined
  427. lovetox_ a windows where history shows
  428. lovetox_ a windows where you can browse history
  429. lovetox_ search filter etc
  430. Ge0rG I never understood that. I want the history in the respective chat tab, with a dynamic filter if possible
  431. lovetox_ i want that too but i have not time to do it 🙂
  432. lovetox_ as you may know Gajim is 15+ years
  433. Ge0rG Yeah.
  434. lovetox_ and dynamic history loading in the same window was not a thing 10 years back
  435. Daniel Also people weirdly love that
  436. Ge0rG which reminds me. Tomorrow is yaxim's 10th birthday.
  437. Ge0rG Daniel: people love the history window?
  438. Daniel Yes
  439. Ge0rG Weird indeed
  440. Holger You just know weird people.
  441. jonas’ most people are weird
  442. lovetox_ also displaying only in the chat window is not a solution for all circumstances
  443. Ge0rG lovetox_: now I'm curious
  444. lovetox_ i may want to look at history with contacts that are not in my roster anymore
  445. lovetox_ or MUCs i dont want to join anymore
  446. lovetox_ having it *only* in the chat has drawbacks
  447. Ge0rG lovetox_: what's wrong with displaying those in the respective chat tabs, without joining?
  448. lovetox_ this means i have to have a concept of all mucs i ever joined, must be able to open them and look at history *without* joining
  449. lovetox_ and how do i know if the user wants to only look at history or want to join=?
  450. lovetox_ etc etc
  451. Ge0rG Yes.
  452. lovetox_ this is of course the easy thing for 90% of all use cases
  453. Daniel And space bar heating
  454. waqas has joined
  455. Ge0rG Yesterday I ran hashcat on my laptop's Intel GPU. There was a frightening smell of something burnt. And space bar heating.
  456. zach has left
  457. zach has joined
  458. morgan has left
  459. lovetox_ has left
  460. lovetox_ has joined
  461. Lance has left
  462. Fahrgast has left
  463. morgan has joined
  464. morgan has left
  465. waqas has left
  466. jabberjocke has joined
  467. jabberjocke_ has joined
  468. Fahrgast has joined
  469. morgan has joined
  470. Lance has joined
  471. jabberjocke_ has left
  472. jabberjocke has left
  473. zach has left
  474. zach has joined
  475. LNJ has joined
  476. morgan has left
  477. pdurbin has joined
  478. lovetox_ has left
  479. morgan has joined
  480. Fahrgast has left
  481. adityaborikar has joined
  482. Fahrgast has joined
  483. aj has joined
  484. neshtaxmpp has left
  485. neshtaxmpp has joined
  486. pdurbin has left
  487. morgan has left
  488. morgan has joined
  489. jmpman has left
  490. morgan has left
  491. lovetox_ has joined
  492. lovetox_ has left
  493. Lance has left
  494. zach has left
  495. zach has joined
  496. lovetox_ has joined
  497. aj has left
  498. lovetox_ Ge0rG : what i save in the table is, archive_jid, last_mam_id, last_timestamp
  499. lovetox_ i save the timestamp because i ran into the following problem
  500. lovetox_ client is turned off 2 weeks, i join a muc and i need to know that the last mam id is 2 weeks back, because i dont want to request that many messages
  501. lovetox_ so i can now switch to a timebased query, like give me last 2 days
  502. lovetox_ i apply this only to public mucs
  503. Ge0rG interesting, thanks
  504. Ge0rG lovetox_: is archive_jid the JID of the MAM service? Do you have your own domain in there for account MAM?
  505. lovetox_ yes
  506. lovetox_ no
  507. jubalh has left
  508. lovetox_ its just the jid of the archive
  509. lovetox_ and yes my own jid is in this table
  510. lovetox_ but i apply different rules to account mam query, i always request everything there, except on very first start
  511. Lance has joined
  512. Ge0rG what do you request on first start?
  513. lovetox_ i think 7 days
  514. lovetox_ but i have a button in gajim, that lets you sync any timeperiod before
  515. zach has left
  516. zach has joined
  517. Ge0rG into the history window? ;)
  518. lovetox_ yes
  519. lovetox_ this is not a button per chat
  520. lovetox_ this is a account button, sync everything, last 6 months, last 3 months
  521. Nekit has left
  522. lovetox_ like one time shot, after setting up a client, do i want to have my whole archive locally or not
  523. lovetox_ users sometimes want to test something and set up a new instance etc, in this case i dont want to download 10.000 messages
  524. Nekit has joined
  525. adityaborikar has left
  526. adityaborikar has joined
  527. jmpman has joined
  528. morgan has joined
  529. morgan has left
  530. zach has left
  531. zach has joined
  532. Lance has left
  533. jabberjocke_ has joined
  534. jabberjocke has joined
  535. Mikaela has left
  536. valo has left
  537. valo has joined
  538. UsL has left
  539. Mikaela has joined
  540. Lance has joined
  541. Nekit has left
  542. Nekit has joined
  543. Mikaela has left
  544. Mikaela has joined
  545. morgan has joined
  546. pdurbin has joined
  547. morgan has left
  548. jubalh has joined
  549. waqas has joined
  550. adityaborikar has left
  551. adityaborikar has joined
  552. morgan has joined
  553. mathieui has left
  554. mathieui has joined
  555. morgan has left
  556. waqas has left
  557. pdurbin has left
  558. murabito has left
  559. jabberjocke has left
  560. zach has left
  561. zach has joined
  562. jmpman has left
  563. jabberjocke has joined
  564. debacle has joined
  565. jabberjocke_ has left
  566. jabberjocke has left
  567. jabberjocke_ has joined
  568. jabberjocke has joined
  569. jabberjocke_ has left
  570. jabberjocke has left
  571. Lance has left
  572. jabberjocke_ has joined
  573. jabberjocke has joined
  574. stpeter has joined
  575. peter has joined
  576. jabberjocke has left
  577. jabberjocke_ has left
  578. peter has left
  579. jabberjocke_ has joined
  580. jabberjocke has joined
  581. jabberjocke_ has left
  582. jabberjocke has left
  583. jabberjocke_ has joined
  584. jabberjocke has joined
  585. jabberjocke has left
  586. jabberjocke_ has left
  587. debacle has left
  588. jabberjocke_ has joined
  589. jabberjocke has joined
  590. jmpman has joined
  591. mr.fister has joined
  592. jabberjocke_ has left
  593. jabberjocke has left
  594. jabberjocke has joined
  595. jabberjocke_ has joined
  596. Lance has joined
  597. stpeter has left
  598. Fahrgast has left
  599. lskdjf has left
  600. lskdjf has joined
  601. zach has left
  602. zach has joined
  603. morgan has joined
  604. Nekit has left
  605. Nekit has joined
  606. morgan has left
  607. zach has left
  608. zach has joined
  609. stpeter has joined
  610. Daniel has left
  611. Daniel has joined
  612. Fahrgast has joined
  613. jabberjocke has left
  614. jabberjocke_ has left
  615. Mikaela has left
  616. Mikaela has joined
  617. Daniel has left
  618. Daniel has joined
  619. jabberjocke_ has joined
  620. jabberjocke has joined
  621. Lance has left
  622. zach has left
  623. zach has joined
  624. stpeter has left
  625. COM8 has joined
  626. COM8 has left
  627. COM8 has joined
  628. COM8 has left
  629. COM8 has joined
  630. COM8 has left
  631. nyco has joined
  632. jabberjocke_ has left
  633. jabberjocke has left
  634. nyco time?
  635. ralphm bangs gavel
  636. Guus I'm just being called
  637. ralphm 0. Welcome + Agenda
  638. Guus I'll lurk for the duration of this call
  639. ralphm Hi all!
  640. COM8 has joined
  641. nyco so we're 2.5?
  642. nyco is it a quorum?
  643. ralphm hehe, not yet
  644. ralphm Seve and MattJ will surely show up
  645. Guus I'm afraid this is going to take a while
  646. Guus customer on the phone
  647. jabberjocke_ has joined
  648. jabberjocke has joined
  649. ralphm Guus: no worries
  650. nyco for a good or a bad news?
  651. COM8 has left
  652. MattJ Hey
  653. nyco cool
  654. ralphm There we go.
  655. Guus unsure
  656. Guus count me out guys, sorry
  657. MattJ Sorry, I honestly didn't know it was Thursday :)
  658. ralphm 1. Minute taker
  659. ralphm MattJ: happens roughly every 7 days
  660. MattJ I'll make a note of that
  661. pdurbin has joined
  662. nyco or every 14 days
  663. jabberjocke_ has left
  664. jabberjocke has left
  665. nyco I'll do the minutes
  666. ralphm Thanks
  667. ralphm I thought MattJ was making notes :-D
  668. ralphm 2. Badges
  669. nyco poll finished, sent to board
  670. Alex has left
  671. ralphm Right
  672. zach has left
  673. ralphm So, my summary: overwhelming support for https://opensourcedesign.net/jobs/jobs/2019-03-19-design-of-badges-for-different-xmpp-compliance-levels
  674. zach has joined
  675. Ge0rG Is it possible to get the poll results without getting elected into next Board?
  676. ralphm And overall enthousiasm for the concept and willingness to use
  677. nyco not a problem for me to send those results to members
  678. ralphm Ge0rG: sorry for the spoiler
  679. madhur.garg has left
  680. madhur.garg has joined
  681. Ge0rG nyco: please do
  682. Ge0rG ralphm: I don't mind, as long as we are making progress here
  683. Lance has joined
  684. stpeter has joined
  685. nyco we have three tickets regardin badges on the trello board, maybe we can archive two
  686. nyco what's next?
  687. ralphm I move we choose mray's design, request the assignment of rights, and then publish the respective badges
  688. Nekit has left
  689. ralphm +1
  690. MattJ +1
  691. nyco +1
  692. nyco who's in charge?
  693. moparisthebest has joined
  694. Nekit has joined
  695. Ge0rG is this about the optical design or also about the content?
  696. ralphm The optical design
  697. Ge0rG yay!
  698. ralphm I tried to find who was involved, and it seems Ge0rG is
  699. ralphm :-D
  700. ralphm (in the 'how to apply' section)
  701. ralphm But I can contact mray, if his contact details on this site are valid.
  702. ralphm I'll make a new card and archive the others.
  703. Ge0rG ralphm: please move on, unless you need some kind of assistance from my side
  704. nyco thx
  705. ralphm Ge0rG: unless you know this person
  706. Ge0rG ralphm: not at all. the only contact we had was my posting of the job
  707. pdurbin has left
  708. ralphm 3. Maxime Buquet => Editor Team
  709. ralphm pep. has requested to be on the Editor team
  710. nyco that would be cool
  711. ralphm Since he's a Member, +1
  712. Seve I'm here, my bad guys, was trapped in a meeting
  713. nyco what's the procedure here?
  714. ralphm we assign him this role, done
  715. nyco +1
  716. nyco —o/ Seve
  717. MattJ +1
  718. pep. And then I request superpowers to the right people?
  719. Seve +1 for Maxime Buquet :)
  720. mr.fister has left
  721. nyco who can grant that?
  722. mr.fister has joined
  723. ralphm Can somebody create a PR for this?
  724. ralphm nyco: we do
  725. ralphm nyco: Board gets to assign people to work teams
  726. nyco I meant superpowers... then I'm interested...
  727. mr.fister has left
  728. mr.fister has joined
  729. MattJ pep., get cape, wear cape, fly!
  730. pep. o/
  731. ralphm Ah, I'm sure jonas’ can help out
  732. nyco destroy Thanos
  733. ralphm or stpeter
  734. lovetox_ has left
  735. ralphm (not the destroying, the giving of powers)
  736. nyco :'(
  737. ralphm 4. Roadmap
  738. ralphm I've started this, and some of this has spilled to this room
  739. ralphm I've found several areas that need work, in my not so humble opinion, which I should probably fold into a nice e-mail
  740. nyco why not a PR?
  741. ralphm Examples include: MIX (including how MAM should work), richer messaging (also touching upon MAM in a fairly big way), and establishing Jingle calls in 2019 (with multiple resources, some of which might be not actually connected)
  742. nyco good
  743. nyco spam?
  744. Ge0rG ralphm: I'm very much interested in a roadmap for the upcoming Compliance Suite 2020, because people have shown interest in the goals we have, and not just the status-quo XEPs
  745. ralphm nyco: because I think that a) the list is not complete, b) I'm sure others have opinions
  746. nyco I meant: should we add spam to our roadmap?
  747. ralphm Ge0rG: the discussion is here on a more general roadmap, not just for the short term, but yes, having an idea of what should be in 2020 is an interesting topic in itself
  748. Seve I haven't followed the room lately on this regard, but stating things that need improvement is a grest step forward, would love to see this happen
  749. Nekit has left
  750. Seve great*
  751. ralphm Ge0rG: but I'm not sure how 2020 could contain stuff that isn't mostly already there in some ways
  752. Nekit has joined
  753. stpeter has left
  754. Ge0rG ralphm: maybe CS-2020 should have a short-term agenda for 2021, and you prepare the long-term agenda?
  755. Ge0rG I don't have any specific ideas yet.
  756. ralphm It could. In any case, whatever both lists will end up being, I hope Council will also have opinions on this, before we say: this is it
  757. Seve Agree
  758. zach has left
  759. zach has joined
  760. jonas’ ralphm, pep., FWIW, someone from iteam with powers on github is needed to grant permissions, I don’t think I have those permissions myself
  761. ralphm Ge0rG: so my idea was sending a mail including those examples, and have a bit of discussion, getting to some common ground
  762. ralphm jonas’: in that case, MattJ or Kev can help
  763. nyco then an email is better than a PR, ok for me
  764. ralphm yay
  765. Ge0rG speaking of iteam, https://mail.jabber.org/pipermail/standards/ is still broken
  766. Ge0rG ralphm: yes, please do that
  767. ralphm Ge0rG: noted. I'll raise it in their room
  768. Ge0rG 👍
  769. ralphm 5. AOB
  770. ralphm ?
  771. ralphm Oh, I noticed that JC no longer has time to do the newsletter
  772. Seve Yes, wanted to talk with him
  773. Seve But basically I would like to discuss with commteam a bit this topic
  774. ralphm And has announced that he's no longer doing that
  775. ralphm Seve: sure, just a thing to mention
  776. ralphm Anything else?
  777. Seve I'm part of it, and would like to see first what are we going to do
  778. Seve Nothing here
  779. ralphm Splendid
  780. ralphm 6. Date of Next
  781. ralphm +1W
  782. nyco I collect links... I wanted to write a bit today, I'll still be in support/help/contrib mode
  783. jmpman has left
  784. ralphm 7. Close
  785. ralphm Thanks all!
  786. ralphm bangs gavel
  787. Ge0rG nyco: also please mail the poll results to members@
  788. Seve Thank you guys, and sorry for my late show up
  789. nyco sir, yes sir
  790. ralphm Better late than never
  791. nyco no pb
  792. nyco thx all
  793. nyco I'm on the minutes, then the poll results to members
  794. jonas’ general remark: I forgot to send out emails for the changes i pushed to xeps on tuesday, I shall do that tonight or at the weekend
  795. ralphm jonas’: thanks!
  796. Ge0rG suppresses a sarcastic email about the email archive not being accessible anyway.
  797. Ge0rG suppresses a sarcastic comment about the email archive not being accessible anyway.
  798. jonas’ well done, Ge0rG!
  799. Mikaela has left
  800. Fahrgast has left
  801. nyco both done
  802. j.r has left
  803. j.r has joined
  804. UsL has joined
  805. j.r has left
  806. Nekit has left
  807. Nekit has joined
  808. j.r has joined
  809. Mikaela has joined
  810. Fahrgast has joined
  811. j.r has left
  812. jmpman has joined
  813. adityaborikar has left
  814. adityaborikar has joined
  815. zach has left
  816. zach has joined
  817. COM8 has joined
  818. j.r has joined
  819. COM8 has left
  820. COM8 has joined
  821. COM8 has left
  822. COM8 has joined
  823. COM8 has left
  824. COM8 has joined
  825. COM8 has left
  826. COM8 has joined
  827. zach has left
  828. zach has joined
  829. COM8 has left
  830. COM8 has joined
  831. COM8 has left
  832. adityaborikar has left
  833. adityaborikar has joined
  834. Lance has left
  835. zach has left
  836. zach has joined
  837. j.r has left
  838. stpeter has joined
  839. peter has joined
  840. Nekit has left
  841. Nekit has joined
  842. UsL has left
  843. Lance has joined
  844. peter has left
  845. COM8 has joined
  846. Nekit has left
  847. Nekit has joined
  848. zach has left
  849. zach has joined
  850. COM8 has left
  851. COM8 has joined
  852. ralphm Nice
  853. COM8 has left
  854. jubalh has left
  855. Fahrgast has left
  856. UsL has joined
  857. sezuan has left
  858. Fahrgast has joined
  859. COM8 has joined
  860. stpeter has left
  861. pdurbin has joined
  862. jmpman has left
  863. COM8 has left
  864. j.r has joined
  865. waqas has joined
  866. zach has left
  867. zach has joined
  868. pdurbin has left
  869. mr.fister has left
  870. Mikaela has left
  871. Mikaela has joined
  872. jmpman has joined
  873. COM8 has joined
  874. waqas has left
  875. COM8 has left
  876. UsL has left
  877. zach has left
  878. zach has joined
  879. morgan has joined
  880. morgan has left
  881. Steve Kille has left
  882. Wojtek has joined
  883. Wojtek has left
  884. morgan has joined
  885. Steve Kille has joined
  886. morgan has left
  887. Nekit has left
  888. Nekit has joined
  889. zach has left
  890. zach has joined
  891. Fahrgast has left
  892. Chobbes has joined
  893. valo has left
  894. valo has joined
  895. morgan has joined
  896. Chobbes has left
  897. morgan has left
  898. waqas has joined
  899. Chobbes has joined
  900. Fahrgast has joined
  901. moparisthebest has left
  902. jmpman has left
  903. LNJ has left
  904. moparisthebest has joined
  905. morgan has joined
  906. zach has left
  907. zach has joined
  908. jubalh has joined
  909. LNJ has joined
  910. jubalh has left
  911. lovetox has joined
  912. morgan has left
  913. waqas has left
  914. Alex has joined
  915. zach has left
  916. zach has joined
  917. mr.fister has joined
  918. Fahrgast has left
  919. Fahrgast has joined
  920. Nekit has left
  921. morgan has joined
  922. Chobbes has left
  923. Steve Kille has left
  924. zach has left
  925. zach has joined
  926. peter has joined
  927. stpeter has joined
  928. jubalh has joined
  929. Fahrgast has left
  930. peter has left
  931. j.r has left
  932. pdurbin has joined
  933. Fahrgast has joined
  934. morgan has left
  935. zach has left
  936. pdurbin has left
  937. zach has joined
  938. waqas has joined
  939. debacle has joined
  940. stpeter has left
  941. adityaborikar has left
  942. waqas has left
  943. waqas has joined
  944. lovetox has left
  945. zach has left
  946. zach has joined
  947. j.r has joined
  948. lovetox has joined
  949. morgan has joined
  950. morgan has left
  951. morgan has joined
  952. zach has left
  953. zach has joined
  954. stpeter has joined
  955. peter has joined
  956. Fahrgast has left
  957. peter has left
  958. morgan has left
  959. morgan has joined
  960. jabberjocke_ has joined
  961. jabberjocke has joined
  962. debacle has left
  963. Fahrgast has joined
  964. morgan has left
  965. morgan has joined
  966. zach has left
  967. zach has joined
  968. jubalh has left
  969. stpeter has left
  970. sezuan has joined
  971. peter has joined
  972. stpeter has joined
  973. peter has left
  974. jubalh has joined
  975. zach has left
  976. zach has joined
  977. jubalh > I'm living between Munich and Stuttgart (South Germany) Not far from me
  978. Douglas Terabyte has left
  979. stpeter has left
  980. eevvoor has joined
  981. Douglas Terabyte has joined
  982. eevvoor has left
  983. Douglas Terabyte has left
  984. Douglas Terabyte has joined
  985. zach has left
  986. zach has joined
  987. LNJ has left
  988. pdurbin has joined
  989. morgan has left
  990. morgan has joined
  991. LNJ has joined
  992. zach has left
  993. zach has joined
  994. lumi has joined
  995. Douglas Terabyte has left
  996. Fahrgast has left
  997. sezuan has left
  998. pdurbin has left
  999. Douglas Terabyte has joined
  1000. UsL has joined
  1001. Nekit has joined
  1002. sezuan has joined
  1003. Yagiza has left
  1004. Yagiza has joined
  1005. j.r has left
  1006. Lance has left
  1007. j.r has joined
  1008. zach has left
  1009. zach has joined
  1010. UsL has left
  1011. reda-alaoui has joined
  1012. sezuan has left
  1013. kokonoe has left
  1014. Fahrgast has joined
  1015. reda-alaoui Hi everyone, I am implementing XEP-0313 Message Archive Management on the server side. Suppose the server supports offline storage (in addition to MAM) and the recipient of the sent message is offline. A message is sent to the offline recipient. Should the message be archived in the recipient archive WHEN: - the message is received by the recipient's server OR - the recipient turns back online
  1016. kokonoe has joined
  1017. Zash The former I think
  1018. Zash Interaction between message archive and offline message store is something that I don't think is fully figured out yet
  1019. reda-alaoui Thank you Zash , but that means that when the recipient client comes back online, it will probably pull the recipient archive containing the archived message while the server will be pushing the offline stored message. How will the client be able to know that the pushed message is the same as the archived message?
  1020. zach has left
  1021. zach has joined
  1022. Zash <stanza-id>
  1023. Alex has left
  1024. reda-alaoui Ok I thought stanza-id were limited to XEP-0313. From my understanding stanza-id is set by the server. How can we provide the stanza-id outside of XEP-0313 IQ queries?
  1025. Fahrgast has left
  1026. Zash The way it goes in Prosody is: - Message comes in - Message is added to the archive - The archive ID is attached to the stanza - Delivery is attempted - If there were no online clients with non-negative priority to accept the stanza, add to offline store (with the stanza-id)
  1027. Zash Not optimal, but it is the way it is because of historical reasons and the thing being modular.
  1028. reda-alaoui So you are adding stanza-id to the offline storage mechanism even if it is not described in https://xmpp.org/extensions/xep-0160.html? Or am I missing a fundamental specification that introduced stanza-id for every message?
  1029. Yagiza has left
  1030. Zash The module that handles MAM adds it when it adds the stanza to the archive. Then it just tags along through the rest of the chain of events.
  1031. morgan has left
  1032. Ge0rG Zash: when do Carbons happen?
  1033. Zash Ge0rG: Somewhere in the middle
  1034. Zash Before normal delivery I think
  1035. Ge0rG reda-alaoui: ideally, you could implement offline storage as a pointer into the MAM archive. When a client fetches from MAM, that pointer is moved forward. Under ideal conditions, you end up with the pointer after the end of the archive and no need to send anything after the MAM sync
  1036. lovetox reda-alaoui, XEP313 says you MUST add the stanza-id to every message that is archived
  1037. Ge0rG There are some corner cases with MAM limited to roster or clients only fetching a subset
  1038. pep. So.. we're stuck on pretty basic stuff on slix while trying to add async functionalities in core parts. How do people even guarantee in-order delivery of stanzas? :x
  1039. lovetox so if you send a message that is in your offline store AND mam archive, but dont attach a stanza-id to the offline message
  1040. lovetox you are violating 313
  1041. Ge0rG pep.: you can't with MAM
  1042. lovetox because now you sent a message without stanza-id that is in fact in the MAM archive
  1043. reda-alaoui lovetox yes I know but I didn't know it should be propagated to other specs
  1044. pep. Ge0rG, not talking about MAM
  1045. mathieui Ge0rG, we’re more in an omemo case
  1046. Ge0rG lovetox: technically, you could have a separate store for offline that's filled before the message gets a MAM ID
  1047. pep. But it's not really omemo related either tbh
  1048. reda-alaoui Ge0rG : that's what we have currently
  1049. Ge0rG mathieui: stupid specs come with stupid consequences for the implementation
  1050. lovetox Ge0rG, it does not matter really, because in the moment you send a message from the offline store, you probably would put it into mam also
  1051. lovetox and then you have to inject a stanza-id anyway
  1052. Ge0rG reda-alaoui: but it helps a client tremendously to deduplicate, if the MAM ID is always sent
  1053. lovetox Ge0rG, its not about help
  1054. lovetox its mandatory
  1055. reda-alaoui Ge0rG yes of course, it is a must have
  1056. reda-alaoui ok, thanks a lot !
  1057. reda-alaoui back to my code then :p
  1058. remko has left
  1059. lovetox thing is a client cant differentiate between a offline message
  1060. lovetox and a live message
  1061. lovetox this means i receive suddenly messages from my mam:2 server that have no stanza-id, this would in best case drop a warning
  1062. lovetox actually just lets not test this :)
  1063. Zash Implementation details galore
  1064. lovetox worst case is, i have to fallback to timestamp/body based deduplication
  1065. lovetox or even worse a client doesnt implement that because he only supports mam:2 and depends on dedup with stanza.id
  1066. Zash Transition periods are messy
  1067. lovetox i have to implement that offline message purge
  1068. Ge0rG lovetox: the paragraph you quoted can be read as "the server must add the ID to the archived message only"
  1069. Daniel pep.: my outgoing stanza queue is synchronized
  1070. lovetox yeah yeah, i know we can read everything the other way, you are right its a grey area that was not thought about when spec mam
  1071. lovetox so yeah please server developer add stanza-id :D
  1072. Daniel So if I need stanzas in order I write them to that queue from the same thread and everything is fine
  1073. pep. Daniel, how do you handle OMEMO encryption for exapmle, where you have to fetch devicelists and bundles and only then send your encrypted message
  1074. Zash Would it be sane that if you set MAM defalut to roster, to then reject anything that doesn't go in the archive when you're offline?
  1075. Zash As in, bounce with an error
  1076. Daniel pep.: call backs
  1077. Ge0rG > servers SHOULD include the element as a child of the forwarded message when using Message Carbons (XEP-0280) Another non-requirement.
  1078. pep. Daniel, Ok so you block sending on the stream?
  1079. zach has left
  1080. pep. Well, yeah it's synchronized
  1081. Steve Kille has joined
  1082. zach has joined
  1083. Fahrgast has joined
  1084. Zash Ge0rG: Make it a MUST for routing 2.0 or whatsitcalled
  1085. LNJ has left
  1086. Daniel No I guess you could send other stanzas while waiting for the omemo fetches to return
  1087. Ge0rG Zash: will you implement that in prosody?
  1088. pep. Daniel, and then you need to keep track of what came in what order, but you also need to prioritize the OMEMO IQs to be able to send that one message
  1089. Zash Ge0rG: I make no promises about what I do with my free time.
  1090. Ge0rG I've just realized that smack will process messages asynchronously, so when I page through 2K of MAM messages, and update the progress counter accordingly, it will arrive at 100% while the async handler is still busy somewhere in the first half
  1091. lovetox update it on receiving the end of the page?
  1092. Daniel pep.: well I guess I can ensure order for a particular set of stanza but I don't have to. For example I can make sure that a leave and a join presence are always send after each other
  1093. Daniel Also the omemo message stanza waiting to be encrypted is not in the queue
  1094. Daniel Yet
  1095. pep. In poezio atm, I am implementing the encrypt method as a stream filter, i.e., I want to catch everything going out to be sure none of what should be encrypted is sent in plain
  1096. j.r has left
  1097. Daniel I send plain text into the omemo pipeline and only after it was encrypted it becomes a stanza and gets fed into the queue
  1098. j.r has joined
  1099. pep. That means I can just send a normal message, and my filter will take care of the encryption
  1100. pep. But that blocks the whole loop
  1101. Daniel > In poezio atm, I am implementing the encrypt method as a stream filter, i.e., I want to catch everything going out to be sure none of what should be encrypted is sent in plain Yeah I'm not a fan of that approach
  1102. Daniel That's what caused the gajim otr html body leak
  1103. pep. hmm?
  1104. lovetox pep. on hitting send, i check if every pre condition is met for encrypting
  1105. pep. The point is to prevent that
  1106. lovetox and if not, just nothing happens
  1107. lovetox text is still in input
  1108. pep. Otherwise I have no guarantee poezio won't send plain stuff behind my back
  1109. UsL has joined
  1110. lovetox i query whatever needs to be queried
  1111. pep. (chatstates, etc.)
  1112. pep. (and other plugins)
  1113. Fahrgast has left
  1114. Daniel pep.: but if I understand you correctly what you are doing is creating a message stanza with a plain text body and then your filter removes the body and inserts omemo. I never do that. An omemo message for me is never a message stanza
  1115. Daniel pep.: but if I understand you correctly what you are doing is creating a message stanza with a plain text body and then your filter removes the body and inserts omemo. I never do that. An omemo message for me is never a message stanza with a normal body
  1116. lovetox Daniel because you dont have a plugin system
  1117. pep. :)
  1118. Daniel Think of stanzas as being immutable.
  1119. pep. Daniel, we're in python
  1120. pep. Nothing is immutable
  1121. Daniel There are not really in my code but I treat them as such
  1122. Daniel You can still think of them as such
  1123. Daniel lovetox: yeah I don't like that kind of plugin systems
  1124. pep. plugin system or not, I have to review every bits of poezio to ensure I won't leak plain. Stream filters were kind of the escape hatch that would allow me not to do that
  1125. jcbrand has left
  1126. pep. plugin system or not, I have to review every bits of poezio to ensure I won't leak plain. Stream filters were kind of the escape hatch that would have allowed me not to do that
  1127. lovetox Daniel what you do you can only do if the client has knowledge about encryption
  1128. Daniel lovetox: yes. Obviously
  1129. mathieui the basic use case ("press enter") is fine, the issue is the rest of the commands and such (/correct, etc) which we have to check and disable
  1130. pep. Also that means it doesn't make sense to implement OMEMO as a plugin in poezio anymore, without stream filters
  1131. lovetox yeah, Gajim has no knowledge about encryption, it just passes the stanza right before sending to the encryption plugin
  1132. pep. And that's not possible license-wise atm
  1133. Nekit has left
  1134. lovetox mathieui, correct is perfectly fine to use with omemo
  1135. mathieui lovetox, yes, but the point is we need to not forget any command which might send plaintext
  1136. lovetox yeah thats a really error prone way of doing this i think
  1137. lovetox instead poezio should not care at all about it
  1138. pep. Which is why I went the stream filter way.. :P
  1139. lovetox do everything with the stanza as it would normally
  1140. lovetox and at the end omemo runs it through a whitelist of stanza nodes
  1141. mathieui lovetox, that was the plan
  1142. Daniel Is there an otr plugin for poezio?
  1143. pep. yeah
  1144. pep. Which is..
  1145. Daniel Because for that you also have to wait
  1146. Daniel Even for remote peers. Which is worse
  1147. pep. I actually have no clue, I won't pronounce myself on it :x
  1148. lovetox but this has nothing to do with the problem pep. described i think
  1149. mathieui Daniel, the OTR plugin is implemented in the way I described above
  1150. lovetox you should have a handler that checks before creating a stanza, even before removing the message from the input field
  1151. lovetox if encryption is even possible
  1152. mathieui which is *really* ugly (whitelist elements, intercept the code that sends messages etc)
  1153. lovetox not pass it to haf of poezio than at the end find out we cant even encrypt because key is missing
  1154. pep. lovetox, "before creating a stanza" doesn't map as you think it does, in slix
  1155. lovetox this has nothing to do with slix
  1156. lovetox that is UI code
  1157. lovetox you press the button
  1158. pep. hmm?
  1159. lovetox on button pres, you check
  1160. Tobias has left
  1161. Daniel > lovetox, "before creating a stanza" doesn't map as you think it does, in slix Then do it in poezio. Not in slix
  1162. pep. nowhere I mentioned UI :P
  1163. pep. I actually don't care about UI for now
  1164. Ge0rG This is the moment when you introduce numeric stanza filter priorities, so that you can do encryption last.
  1165. pep. Daniel, yeah, slix/poezio ~
  1166. Zash And then you need something *really last*
  1167. Ge0rG Zash: thus numeric.
  1168. lovetox pep., you check before the message would vanish from the input
  1169. Zash priority=infinity+1
  1170. Ge0rG And then a plugin author thinks they must be laster than last and change the body post encryption.
  1171. Daniel Did I mention I don't like plug-ins?
  1172. Zash Double-edged sword if anything
  1173. lovetox also writing a client is one thing
  1174. lovetox writing a client that perfectly interfaces with all kind of plugins
  1175. lovetox a whole other game :)
  1176. Ge0rG Just write a plugin management system and let the modules do the rest.
  1177. kokonoe has left
  1178. Ge0rG If you are okay with Lua, there is a decent plugin management system that's handling xml streams already.
  1179. Zash Ge0rG: You know how Prosody is actually a Lua variant of node.js with some plugin management on top?
  1180. Ge0rG Zash: no.
  1181. Zash Unheard of!
  1182. Ge0rG Zash: prosody requires a VirtualHost, so it can't be what you claim it is.
  1183. pep. Daniel, yeah that's a different discussion, I don't have a choice anymore :)
  1184. kokonoe has joined
  1185. Fahrgast has joined
  1186. Daniel In any case I suppose you currently have something of a `on_message` hook that has a message stanza as a parameter. What you probably want to do is create another hook `on_message_submitted` that has the plain text of the input and gets fired as soon as the user presses enter.
  1187. zach has left
  1188. zach has joined
  1189. Daniel And if that hook returns true it gets into the normal pipeline and the input field gets cleared
  1190. Daniel And for omemo message you handle it differently
  1191. Ge0rG But then you need an internal representation for all message types, which is then mapped to the respective xml on transmission.
  1192. Daniel And then after omemo ran over it you can re-emit the message stanza and run it though the chain
  1193. pep. Daniel, yeah, it's already what I'm doing but at the slix level. I'd need to look into it because atm it seems to me I'm going to run into the same problems
  1194. Daniel Same problem = leaking stuff you don't want to leak?
  1195. pep. no
  1196. pep. I lose in-order guarantees
  1197. pep. Either that or I block my stream for ages while fetching devicelists and bundles
  1198. pep. https://lab.louiz.org/poezio/slixmpp/merge_requests/24/diffs that's supposed to help here, but it's still not what we want
  1199. peter has joined
  1200. peter has left
  1201. Chobbes has joined
  1202. alameyo has left
  1203. alameyo has joined
  1204. Alex has joined
  1205. jubalh has left
  1206. Lance has joined
  1207. madhur.garg has left
  1208. madhur.garg has joined
  1209. wurstsalat has left
  1210. Chobbes has left
  1211. Chobbes has joined
  1212. zach has left
  1213. zach has joined
  1214. Fahrgast has left
  1215. Chobbes has left
  1216. Chobbes has joined
  1217. UsL has left
  1218. Fahrgast has joined
  1219. stpeter has joined
  1220. peter has joined
  1221. karoshi has left
  1222. kokonoe has left
  1223. kokonoe has joined
  1224. lovetox has left
  1225. Steve Kille has left
  1226. Chobbes has left
  1227. goffi has left
  1228. Mikaela has left
  1229. zach has left
  1230. zach has joined
  1231. pdurbin has joined
  1232. lskdjf has left
  1233. larma has left
  1234. kokonoe has left
  1235. kokonoe has joined
  1236. reda-alaoui has left
  1237. pdurbin has left
  1238. Allo has left
  1239. Allo has joined
  1240. peter has left
  1241. lumi has left
  1242. Zash has left