XSF Discussion - 2019-08-22

  1. stpeter has left

  2. Fahrgast has left

  3. Daniel has left

  4. Daniel has joined

  5. Fahrgast has joined

  6. zach has left

  7. zach has joined

  8. jmpman has left

  9. Daniel has left

  10. Fahrgast has left

  11. Allo has left

  12. Fahrgast has joined

  13. Daniel has joined

  14. Allo has joined

  15. Daniel has left

  16. Daniel has joined

  17. UsL has left

  18. UsL has joined

  19. aj has joined

  20. stpeter has joined

  21. peter has joined

  22. adityaborikar has joined

  23. aj has left

  24. patrick has left

  25. zach has left

  26. zach has joined

  27. Chobbes has left

  28. adityaborikar has left

  29. adityaborikar has joined

  30. pdurbin has joined

  31. Daniel has left

  32. adityaborikar has left

  33. marc_ has left

  34. Daniel has joined

  35. pdurbin has left

  36. marc_ has joined

  37. neshtaxmpp has left

  38. neshtaxmpp has joined

  39. karoshi has joined

  40. peter has left

  41. zach has left

  42. zach has joined

  43. pdurbin has joined

  44. stpeter has left

  45. adityaborikar has joined

  46. Daniel has left

  47. Daniel has joined

  48. adityaborikar has left

  49. jmpman has joined

  50. zach has left

  51. zach has joined

  52. zach has left

  53. zach has joined

  54. Daniel has left

  55. Fahrgast has left

  56. Fahrgast has joined

  57. Daniel has joined

  58. zach has left

  59. zach has joined

  60. Lance has joined

  61. adityaborikar has joined

  62. jmpman has left

  63. tom has joined

  64. marc_ has left

  65. jmpman has joined

  66. andy has joined

  67. adityaborikar has left

  68. marc_ has joined

  69. Yagiza has joined

  70. adityaborikar has joined

  71. waqas has left

  72. zach has left

  73. zach has joined

  74. Daniel has left

  75. Daniel has joined

  76. Fahrgast has left

  77. zach has left

  78. zach has joined

  79. Lance has left

  80. pdurbin has left

  81. Tobias has joined

  82. pdurbin has joined

  83. Fahrgast has joined

  84. jmpman has left

  85. Lance has joined

  86. Daniel has left

  87. Daniel has joined

  88. Nekit has joined

  89. aj has joined

  90. Daniel has left

  91. wurstsalat has joined

  92. Daniel has joined

  93. Lance has left

  94. zach has left

  95. zach has joined

  96. jabberjocke has left

  97. goffi has joined

  98. Lance has joined

  99. murabito has left

  100. murabito has joined

  101. jonas’

    what has time to do with that?

  102. adityaborikar has left

  103. zach has left

  104. zach has joined

  105. Daniel

    Yes you need to start mam or get the last if *before* you send presence

  106. Daniel

    Yes you need to start mam or get the last ID *before* you send presence

  107. Daniel

    Don't think carbons have anything to do with that

  108. jonas’

    you could enable carbons before sending presence, too

  109. Tobias has left

  110. Tobias has joined

  111. jonas’

    but don’t you need to to MAM *after* sending presence/enabling carbons to not lose a message?

  112. jonas’

    otherwise, if you got the last ID / queried MAM *before* sending presence, a message might’ve arrived between querying the ID and sending presence/enabling carbons, which you’d then miss

  113. Daniel

    Mhhh ideally then you'd get the last ID before presence but fire the query after

  114. jonas’


  115. jonas’

    no, wit

  116. jonas’


  117. jonas’

    if you get the last ID before presence, what happens if a message arrives between you getting the last ID and you sending presence?

  118. Daniel

    jonas’: last ID defines the lower end

  119. jonas’

    what does "lower" mean in that context?

  120. Daniel

    Upper end is undefined

  121. jonas’

    is last ID the last thing you saw in your last session?

  122. jonas’

    or is last ID the newest ID?

  123. Daniel


  124. Daniel

    The newest you currently have locally

  125. ralphm

    But you won't get newer messages until you sent presence.

  126. jonas’

    Daniel, I see!

  127. jonas’

    that makes sense then

  128. jonas’

    it’s early in the morning, sorry :)

  129. sezuan has joined

  130. ralphm

    Daniel: if you're saying there's a specific order of events that ensures you get all messages, is that documented somewhere?

  131. david has left

  132. ralphm

    Like: get last id, send carbons request, send presence, request mam?

  133. ralphm

    Because XEP-0313 says client sync is it of scope (!), and refers to bind2.

  134. ralphm

    out of scope

  135. Ge0rG

    ralphm: an attempt was made to document it in 0313, but it turned out not to work.

  136. Daniel

    what didn’t work?

  137. Daniel

    the attempt?

  138. Ge0rG

    Daniel: the actual ordering that was written down, I think

  139. ralphm

    Ge0rG: so is there an order that does work? Or can it not be solved with current protocols?

  140. Daniel

    the order that ralphm just wrote down doesn’t work?

  141. Ge0rG

    So you note the last id you know, send presence, and then fetch MAM in parallel to receiving offline messages / live traffic? Is there any change to sync linearly in the actual message order?

  142. ralphm

    I believe that the order I mentioned is at-least-once. There's no such thing as exactly-once in general.

  143. Daniel

    well receiving MAM and offline messages in parallel isn’t that bad; you can dedup them

  144. Holger

    Right, there's no way to avoid dedup.

  145. Daniel

    however when i detect that MAM preferences are set to always and server has offline msg mangament things i also run a purge

  146. Daniel

    before sending presence

  147. Holger

    … which can reduce the number of messages to dedup, yes.

  148. Daniel

    but i see that only as a minor traffic optimization

  149. Ge0rG

    You also need to dedup between the Carbons of live messages that arrived between you enabling Carbons and you sending presence, and their respective offline copy

  150. ralphm

    Daniel: is the filter for messages going to MAM and offline identical?

  151. Ge0rG

    ralphm: probably not

  152. Daniel

    my dedup would work fine; and does on servers that don’t have the purge thing

  153. Daniel

    ralphm, i guess there is nothing in the protocol to ensure that

  154. ralphm

    Ge0rG: you don't get carbons until you send presence

  155. Ge0rG

    Daniel: if you purge offline, isn't there a chance that you'll purge too much?

  156. Ge0rG

    ralphm: are you sure?

  157. Daniel

    Ge0rG, well my reasoning is if it doesn’t end up in MAM it wasn’t important

  158. Daniel

    your second client wouldn’t receive it either

  159. ralphm

    That's the idea. Before sending presence, you only get message stanzas directed at your full JID.

  160. Daniel

    i mean if the rules between offline messages and mam differ it is probably time to reconsider the rules

  161. Ge0rG

    ralphm: I don't see presence mentioned in 0280: > When the server receives a <message/> eligible for carbons delivery addressed to a client JID (either bare or full), it delivers the <message/> according to RFC 6121 § 8.5.3, and then delivers a forwarded copy to each Carbons-enabled resource for the matching bare JID recipient that did not receive it under the RFC 6121 delivery rules.

  162. Daniel

    but if you hate the purge don’t purge

  163. Daniel

    that's not something i would codify anywhere

  164. Daniel

    especially since purge capability isn’t really widespread anywhere

  165. ralphm

    Ge0rG: hmm, that surprises me. Curious how servers implement this then.

  166. Steve Kille has left

  167. zach has left

  168. zach has joined

  169. jonas’

    I’m pretty sure that some people use carbons-enabled clients with negative priority to get all messages (also) on a non-carbons-enabled client :)

  170. Ge0rG

    Daniel: I remember complaining about a race condition there, but I can't say right now what it was

  171. Ge0rG

    jonas’: I'm doing that, but a negative presence still is a presence

  172. ralphm


  173. ralphm

    Just for my information, though, the only upshot of that combination (negative presence and carbons) is that you don't get message stanzas to bare JID that don't match the carbon criteria?

  174. ralphm

    I'm curious in what situation that is beneficial.

  175. jonas’

    ralphm, the non-carbons-enabled client will get all the messages because it’s the only one with >= 0 priority

  176. jonas’

    while the carbons-enabled clients also get the messages because they’ve enabled carbons

  177. ralphm

    But only of type chat, for example.

  178. Holger

    So just a trick to get carbons behavior with n clients although only n-1 clients support carbons.

  179. sezuan has left

  180. lovetox_ has joined

  181. jonas’

    "good enough"

  182. ralphm


  183. Daniel

    not that implementing carbons in a client is hard…

  184. ralphm

    It is interesting though. Most clients use presence priority to figure out which 'show' to display.

  185. ralphm

    So in this setting that might mislead

  186. ralphm

    Obscure edge case, maybe

  187. Holger


  188. sezuan has joined

  189. ralphm

    Any way, I think that servers will still wait for presence to start sending any messages.

  190. ralphm

    (that were not to this connection's resource)

  191. Lance has left

  192. waqas has joined

  193. Steve Kille has joined

  194. lovetox_

    It does not matter in what order you activate carbons, presence, offline messages

  195. lovetox_

    you can never miss a message if you do MAM last

  196. lovetox_

    thats the easy way, that might have here and there some message that has to be deduped

  197. lovetox_

    but thats no problem with stanza.-id

  198. jubalh has joined

  199. ralphm

    lovetox_: what was said is that before sending all that, you need to establish the last id you have.

  200. murabito has left

  201. murabito has joined

  202. ralphm

    The rest of the order is less relevant, although there might be an optimum to minimize traffic of dupes.

  203. jcbrand has joined

  204. lovetox_

    you have the last ID in your database

  205. zach has left

  206. zach has joined

  207. ralphm

    Yes, you just need to make sure you don't invalidate it by getting new messages before doing MAM

  208. lovetox_

    yes of course, before MAM catchup is not complete you should not record "last stanza id"

  209. lovetox_

    and MAM catchup will always give you the very last message

  210. ralphm

    That was the thing we tried to convey

  211. ralphm

    I'm still curious as to why it is said to be out of scope and mentions bind2 as possible remedy.

  212. flow

    So is it guaranteed that servers don't mix in live messages while sending MAM messages?

  213. ralphm

    flow: I don't think so

  214. ralphm

    That seems complex to implement

  215. Lance has joined

  216. jonas’

    ralphm, I think the idea was to have a single sync point where you get the ID of the most recent message in your MAM (or possible multiple IDs based on the sender JID) and enable carbons and message delivery in a single server-atomic operation

  217. jonas’

    removing the need to deduplicate there

  218. waqas has left

  219. Ge0rG

    Yes, essentially you need an atomic operation where you enable live traffic and receive the ID of the last non live message

  220. Ge0rG

    So that you can fill the hole.

  221. Ge0rG

    But there is no way to actually receive what you need in a linear order.

  222. jonas’


  223. jmpman has joined

  224. david has joined

  225. david has left

  226. Ge0rG

    and there is no way to receive the messages exactly once.

  227. lovetox_

    flow why would it make a difference if a server mixes in live messages?

  228. jonas’

    Ge0rG, with bind2, there would be, or?

  229. Ge0rG

    jonas’: maybe

  230. jonas’


  231. Ge0rG

    jonas’: if bind2 also removes all MAMed messages from the offline history sync

  232. jonas’

    in my world, offline messages do not exist and you always purge them.

  233. david has joined

  234. ralphm

    jonas’: that sync point seems to be when the server receives presence

  235. jonas’

    ralphm, it’s not a sync point, because you don’t get the "newest MAM ID" from that

  236. jonas’

    (you = the client)

  237. ralphm

    It is a sync point for live messages

  238. jonas’

    how is it sync if it’s one-way?

  239. ralphm

    Well, sure you still need to do the mam query afterwards. I don't see how one can technically achieve exactly once.

  240. jonas’

    ralphm, if the server replied (to the presence, or to whatever) with the ID of the most-recent stanza in the MAM archive at the time the presence was received

  241. jonas’

    then you’d query from "last stanza ID I’ve seen" until "most-recent stanza ID I got from presence"

  242. lovetox_

    you dont record that jonas

  243. zach has left

  244. zach has joined

  245. jonas’

    lovetox_, ?

  246. lovetox_

    you dont record last-stanza-id until you made your mam query

  247. lovetox_

    and while the mam query is running you ignore all stanza-ids from live and carbons

  248. ralphm

    From a client perspective they'd be ideal. I don't think that's easy to implement server side. Also, exactly once delivery is theoretically impossible.

  249. lovetox_

    you only look at the mam query

  250. jonas’

    ralphm, sure.

  251. lovetox_

    the last message in the mam query, is your most recent message, then you record the stanza-id

  252. lovetox_

    and then you are fully synced

  253. lovetox_

    and it does not matter what the server sends you inbetween your mam query

  254. jonas’

    if you don’t want to be efficient, sure.

  255. ralphm

    I've also seen MAM implementations that were eventually consistent, by the way.

  256. lovetox_

    yes jonas thats what we were saying

  257. ralphm

    (server side)

  258. lovetox_

    and nobody needs to be efficient here, mam is complex enough

  259. lovetox_

    a few duplicates are not bad, its totally fine

  260. lovetox_

    and you can now overengineer that so there will never be a message twice, but its not an "issue" of clients that we are so inefficient because we get 3 offline messages while mam query

  261. lovetox_

    and btw you have the same thing with MUC

  262. lovetox_

    live messages will arrive while you query mam

  263. ralphm

    Oh, don't get me started on MUC or MIX.

  264. lovetox_

    i would argue this was a problem back in mam:0 mam:1 days

  265. lovetox_

    if you could not depend on stanza-id

  266. lovetox_

    now with mam:2, really i dont care about a handful of duplicates in a mam catchup

  267. ralphm


  268. ralphm

    Problem with group chats is that you have to do it for each room.

  269. ralphm

    And although MIX currently says you should also store in your local archive (which I like, as it decreases client complexity), it is unclear how a server catches up to the room archive in case of delivery issues (e.g. s2s outages).

  270. Mikaela has joined

  271. Nekit has left

  272. morgan has joined

  273. Mikaela has left

  274. lovetox_

    ralphm what is the problem with making a query to each MUC? inefficient?

  275. Mikaela has joined

  276. lovetox_

    yes maybe i can spare 20 stanzas, for 20 muc joins, but other than that, this just works

  277. lovetox_

    if i switch now to some other concept, like you just decribe and for that get real problems like s2s catchup etc

  278. lovetox_

    i would stay with the old behaviour

  279. murabito has left

  280. murabito has joined

  281. Daniel

    Oh I didn't know that the server was supposed to catch up on your behalf. I thought you were just going to miss messages

  282. Daniel

    Or magically know where the gaps are

  283. morgan has left

  284. remko has joined

  285. Nekit has joined

  286. Ge0rG

    lovetox_: on mobile clients, it makes sense to reduce: round-trips, wakeups, data bandwidth

  287. lovetox_

    yes, i have to trust you on that

  288. lovetox_

    though i would say mobile clients with stream management should not run to often into the, total connection reset scenario

  289. zach has left

  290. zach has joined

  291. Daniel

    Ge0rG, well you'd do mam and join at the same time. so it's not more roundtrips or more wake ups

  292. Daniel

    all it does it redcue traffic

  293. Daniel

    which is fair

  294. ralphm

    Daniel: well, let's say it is unclear. The reason I dislike having to check each channel for its archives, is that it doesn't scale. If you want to implement something like Slack, you can easily be in 40+ channels. In WhatsApp, my daughter is in uncountable ones that are never left.

  295. Ge0rG

    ralphm: yes, we need a way to start a session by just telling the server "my last local ID is X, give me everything relevant after that"

  296. morgan has joined

  297. pdurbin has left

  298. lovetox_ has left

  299. Daniel

    ralphm, depends on what you want to scale and how it is implemented server side. send a message to a group of 500 people - you might not want to store it 501 times in different MAM archives

  300. Daniel

    especially not in 501 transactions

  301. ralphm

    Daniel: I strongly believe that the collective archive of all my conversations should be as close to me as possible. I.e. my own server.

  302. ralphm

    I'm sure there are server side optimizations that could be done.

  303. Ge0rG

    Daniel: having 501 archives wasn't a problem for the designers of MIX

  304. Daniel

    Ge0rG, yes we are talking about mix

  305. Daniel

    (or muc/sub which does similar things for that matter)

  306. lovetox_ has joined

  307. ralphm

    I mean we are also sending 501 copies over the wire.

  308. Nekit has left

  309. Daniel

    ralphm, i get that. i'm with you on that. however if we don’t change how MAM is currently implemented it will trigger 501 transactions

  310. Ge0rG

    Daniel: I think the only actual efficiency problem about MIX is that you end up with 501 copies of each message, with 490 accounts being abandoned by their owner

  311. ralphm

    Even if participants are on the same server. And not counting c2s copies

  312. Daniel

    which is actually a problem we are running into on one project

  313. moparisthebest has left

  314. Ge0rG

    Daniel: a server implementing both the MIX and the account domain could use the same archive table, and just remember for each account when it last joined

  315. ralphm

    If the problem is retention, then there are two ways: limit in time/space, or pay for storage.

  316. Ge0rG

    speaking of MAM, where are MUC-PMs stored?

  317. Daniel

    for us it was not necessary the storage but the throughput

  318. Daniel

    501 transactions are expensive

  319. Ge0rG is still struggling with the right SQL query to obtain the last-stanza-ID for MAM sync

  320. ralphm

    Daniel: did you see my 'eventually consistent' remark above?

  321. morgan has left

  322. Daniel

    ralphm, your point being we just have to do it anyway?

  323. lovetox_

    Ge0rG on your account MAM

  324. lovetox_

    if this was the question

  325. Ge0rG

    lovetox_: yes

  326. lovetox_

    only type=groupchat is in the MUC archive, usually on current servers

  327. ralphm

    Daniel: I'd prefer to have one, complete, archive per user, and only use room archives to catch up with pre-join history, but I'm not sure if this is possible. It would surely involve backfilling, and I think that's at the very least undefined, and impossible at worst, with the way MAM is designed right now.

  328. Daniel

    yes you get your PMs from your normal catchup

  329. morgan has joined

  330. Nekit has joined

  331. Daniel

    ralphm, yes to all that

  332. Dele (Mobile) has joined

  333. Daniel

    and also rather expensive with the way MAM is designed right now

  334. Holger

    Availability of room history is deemed critical enough to dive into the headaches of maintaining local copies even though availibility of the room service itself isn't considered critical enough to distribute it across domains?

  335. Lance has left

  336. Daniel

    did someone say matrix?

  337. Holger

    So we get the cons of Matrix without the pros?

  338. Holger


  339. waqas has joined

  340. Holger

    Not sure we need to agree on a model though. I guess local MIX archives can remain an optional feature without introducing trouble?

  341. lskdjf has joined

  342. Daniel

    as long as it is advertised and clients know what to do

  343. morgan has left

  344. lovetox_

    Holger just out of interest how many messages does the conversations.im server MAM archive contain currently ?

  345. Daniel


  346. lovetox_

    i think you have no MAM storage time limit on that server if i remember correctly

  347. Daniel

    we do; because we ran out of disk space :-)

  348. lovetox_

    haha 😃

  349. larma has joined

  350. Holger

    This came totally unexpected.

  351. Nekit has left

  352. Nekit has joined

  353. adityaborikar has joined

  354. Dele (Mobile) has left

  355. ralphm

    Holger: sure, Matrix solves some these issues, but I believe they haven't tackled group scalability fully yet, and if we're talking about possibly having dimensions in MAM archives for reactions, edits, and other attachments to messages, I am not sure it handles that well at all.

  356. zach has left

  357. zach has joined

  358. Yagiza has left

  359. Yagiza has joined

  360. ralphm

    Having just a dag of commits is not great in that respect

  361. Ge0rG

    couldn't an account server "cheat" by just forwarding all MAM queries to the MIX service?

  362. jonas’

    Ge0rG, only if you were to filter by the MIX service

  363. jonas’

    my understanding is that a normal MAM query would give you MIX and non-MIX messages interleaved

  364. ralphm

    Ge0rG: that might not complete in a reasonable time, and still leave you with gaps?

  365. jmpman has left

  366. ralphm

    What jonas’ says

  367. Daniel

    also how would your server know the relevant stanza-id of the mix service

  368. Ge0rG


  369. Ge0rG

    Right. So many problems.

  370. ralphm

    As I said, this is not easy

  371. ralphm

    By the way, somebody remarked it being ok to send MAM along with join, but join is a one time thing in MIX. After that your server will just send presence.

  372. Ge0rG

    that remark was about MUC

  373. ralphm


  374. Ge0rG

    If only we had threads.

  375. morgan has joined

  376. ralphm


  377. waqas has left

  378. adityaborikar has left

  379. jonas’

    is this the "Can of Worms" day?

  380. Ge0rG

    So to sum it up: Carbons and MAM are ugly workarounds to make an unreliable message "delivery" protocol more reliable, and MIX is codifying those workarounds without solving the actual reliability problems?

  381. Ge0rG

    jonas’: like every day.

  382. Nekit has left

  383. ralphm

    Ge0rG: maybe.

  384. Lance has joined

  385. Ge0rG

    So I need to have a flag for type=groupchat messages, to exclude those from the last-id-query, while retaining other-typed messages from MUC JIDs, like invitation and invitation rejection.

  386. Nekit has joined

  387. Daniel


  388. Ge0rG

    And I need a DB index on that flag.

  389. Daniel

    or do the gajim where you store the last mam ids seperatly

  390. Ge0rG

    What could possibly go wrong with that?

  391. Daniel

    probably the same amount of things that can go wrong with your approach

  392. Daniel

    only different things

  393. Ge0rG

    I'm still wondering, given a set of IDs, how do I determine which one is last?

  394. Ge0rG

    The anticipated answer is: I can't, I need to know from the context

  395. zach has left

  396. zach has joined

  397. Daniel

    the gajim way(tm) also gives you the ability to remove messages locally while still making the catchup work

  398. Daniel

    and not re-fetch a deleted message

  399. aj has left

  400. Daniel

    there are forked versions of Conversations that do the gajim for that purpose

  401. morgan has left

  402. Ge0rG

    Yeah, that makes some sense.

  403. lovetox_

    Ge0rG i would suggest to hold the last stanza id per jid, in a separate table

  404. lovetox_

    and yes stanza-id has no order

  405. Ge0rG

    lovetox_: is there anything else one should put into that table?

  406. lovetox_

    so just from the id you cant determine if its the highest or lowest

  407. lovetox_

    its opaque

  408. Ge0rG

    lovetox_: you update that table when you finish a MAM query?

  409. lovetox_

    no on every MAM message

  410. lovetox_

    so if the catchup breaks up or my client crashes

  411. Ge0rG

    lovetox_: on every in-order MAM message? Or are all your MAM messages in-order?

  412. lovetox_

    i dont have to refetch

  413. Ge0rG

    Sigh. I wish MAM would go away and server would just track the last delivered message per client resource.

  414. Ge0rG

    it would be so easy with 0198, at least for the sake of full-sync clients

  415. lovetox_

    i contact you later Ge0rG with the db schema and what i store, must go now

  416. Ge0rG

    lovetox_: thanks!

  417. flow

    lovetox_> flow why would it make a difference if a server mixes in live messages? I guess it depends if a live message triggers something that touches the processing of the incoming MAM archive messages. I was just wondering what people think about that

  418. ralphm

    For me Gajim never seems to properly retrieve / show MUC history after coming online.

  419. Ge0rG

    What's the status of the Badges poll, BTW? I'd _really_ like to know the results.

  420. morgan has joined

  421. Daniel has left

  422. Daniel has joined

  423. lovetox_

    ralphm this works. but it only shows what you missed

  424. lovetox_

    so if you join twice, all history messages are in the history window CTRL + H

  425. Ge0rG

    what is a history window and why should it even be there?

  426. jmpman has joined

  427. lovetox_

    a windows where history shows

  428. lovetox_

    a windows where you can browse history

  429. lovetox_

    search filter etc

  430. Ge0rG

    I never understood that. I want the history in the respective chat tab, with a dynamic filter if possible

  431. lovetox_

    i want that too but i have not time to do it 🙂

  432. lovetox_

    as you may know Gajim is 15+ years

  433. Ge0rG


  434. lovetox_

    and dynamic history loading in the same window was not a thing 10 years back

  435. Daniel

    Also people weirdly love that

  436. Ge0rG

    which reminds me. Tomorrow is yaxim's 10th birthday.

  437. Ge0rG

    Daniel: people love the history window?

  438. Daniel


  439. Ge0rG

    Weird indeed

  440. Holger

    You just know weird people.

  441. jonas’

    most people are weird

  442. lovetox_

    also displaying only in the chat window is not a solution for all circumstances

  443. Ge0rG

    lovetox_: now I'm curious

  444. lovetox_

    i may want to look at history with contacts that are not in my roster anymore

  445. lovetox_

    or MUCs i dont want to join anymore

  446. lovetox_

    having it *only* in the chat has drawbacks

  447. Ge0rG

    lovetox_: what's wrong with displaying those in the respective chat tabs, without joining?

  448. lovetox_

    this means i have to have a concept of all mucs i ever joined, must be able to open them and look at history *without* joining

  449. lovetox_

    and how do i know if the user wants to only look at history or want to join=?

  450. lovetox_

    etc etc

  451. Ge0rG


  452. lovetox_

    this is of course the easy thing for 90% of all use cases

  453. Daniel

    And space bar heating

  454. waqas has joined

  455. Ge0rG

    Yesterday I ran hashcat on my laptop's Intel GPU. There was a frightening smell of something burnt. And space bar heating.

  456. zach has left

  457. zach has joined

  458. morgan has left

  459. lovetox_ has left

  460. lovetox_ has joined

  461. Lance has left

  462. Fahrgast has left

  463. morgan has joined

  464. morgan has left

  465. waqas has left

  466. jabberjocke has joined

  467. jabberjocke_ has joined

  468. Fahrgast has joined

  469. morgan has joined

  470. Lance has joined

  471. jabberjocke_ has left

  472. jabberjocke has left

  473. zach has left

  474. zach has joined

  475. LNJ has joined

  476. morgan has left

  477. pdurbin has joined

  478. lovetox_ has left

  479. morgan has joined

  480. Fahrgast has left

  481. adityaborikar has joined

  482. Fahrgast has joined

  483. aj has joined

  484. neshtaxmpp has left

  485. neshtaxmpp has joined

  486. pdurbin has left

  487. morgan has left

  488. morgan has joined

  489. jmpman has left

  490. morgan has left

  491. lovetox_ has joined

  492. lovetox_ has left

  493. Lance has left

  494. zach has left

  495. zach has joined

  496. lovetox_ has joined

  497. aj has left

  498. lovetox_

    Ge0rG : what i save in the table is, archive_jid, last_mam_id, last_timestamp

  499. lovetox_

    i save the timestamp because i ran into the following problem

  500. lovetox_

    client is turned off 2 weeks, i join a muc and i need to know that the last mam id is 2 weeks back, because i dont want to request that many messages

  501. lovetox_

    so i can now switch to a timebased query, like give me last 2 days

  502. lovetox_

    i apply this only to public mucs

  503. Ge0rG

    interesting, thanks

  504. Ge0rG

    lovetox_: is archive_jid the JID of the MAM service? Do you have your own domain in there for account MAM?

  505. lovetox_


  506. lovetox_


  507. jubalh has left

  508. lovetox_

    its just the jid of the archive

  509. lovetox_

    and yes my own jid is in this table

  510. lovetox_

    but i apply different rules to account mam query, i always request everything there, except on very first start

  511. Lance has joined

  512. Ge0rG

    what do you request on first start?

  513. lovetox_

    i think 7 days

  514. lovetox_

    but i have a button in gajim, that lets you sync any timeperiod before

  515. zach has left

  516. zach has joined

  517. Ge0rG

    into the history window? ;)

  518. lovetox_


  519. lovetox_

    this is not a button per chat

  520. lovetox_

    this is a account button, sync everything, last 6 months, last 3 months

  521. Nekit has left

  522. lovetox_

    like one time shot, after setting up a client, do i want to have my whole archive locally or not

  523. lovetox_

    users sometimes want to test something and set up a new instance etc, in this case i dont want to download 10.000 messages

  524. Nekit has joined

  525. adityaborikar has left

  526. adityaborikar has joined

  527. jmpman has joined

  528. morgan has joined

  529. morgan has left

  530. zach has left

  531. zach has joined

  532. Lance has left

  533. jabberjocke_ has joined

  534. jabberjocke has joined

  535. Mikaela has left

  536. valo has left

  537. valo has joined

  538. UsL has left

  539. Mikaela has joined

  540. Lance has joined

  541. Nekit has left

  542. Nekit has joined

  543. Mikaela has left

  544. Mikaela has joined

  545. morgan has joined

  546. pdurbin has joined

  547. morgan has left

  548. jubalh has joined

  549. waqas has joined

  550. adityaborikar has left

  551. adityaborikar has joined

  552. morgan has joined

  553. mathieui has left

  554. mathieui has joined

  555. morgan has left

  556. waqas has left

  557. pdurbin has left

  558. murabito has left

  559. jabberjocke has left

  560. zach has left

  561. zach has joined

  562. jmpman has left

  563. jabberjocke has joined

  564. debacle has joined

  565. jabberjocke_ has left

  566. jabberjocke has left

  567. jabberjocke_ has joined

  568. jabberjocke has joined

  569. jabberjocke_ has left

  570. jabberjocke has left

  571. Lance has left

  572. jabberjocke_ has joined

  573. jabberjocke has joined

  574. stpeter has joined

  575. peter has joined

  576. jabberjocke has left

  577. jabberjocke_ has left

  578. peter has left

  579. jabberjocke_ has joined

  580. jabberjocke has joined

  581. jabberjocke_ has left

  582. jabberjocke has left

  583. jabberjocke_ has joined

  584. jabberjocke has joined

  585. jabberjocke has left

  586. jabberjocke_ has left

  587. debacle has left

  588. jabberjocke_ has joined

  589. jabberjocke has joined

  590. jmpman has joined

  591. mr.fister has joined

  592. jabberjocke_ has left

  593. jabberjocke has left

  594. jabberjocke has joined

  595. jabberjocke_ has joined

  596. Lance has joined

  597. stpeter has left

  598. Fahrgast has left

  599. lskdjf has left

  600. lskdjf has joined

  601. zach has left

  602. zach has joined

  603. morgan has joined

  604. Nekit has left

  605. Nekit has joined

  606. morgan has left

  607. zach has left

  608. zach has joined

  609. stpeter has joined

  610. Daniel has left

  611. Daniel has joined

  612. Fahrgast has joined

  613. jabberjocke has left

  614. jabberjocke_ has left

  615. Mikaela has left

  616. Mikaela has joined

  617. Daniel has left

  618. Daniel has joined

  619. jabberjocke_ has joined

  620. jabberjocke has joined

  621. Lance has left

  622. zach has left

  623. zach has joined

  624. stpeter has left

  625. COM8 has joined

  626. COM8 has left

  627. COM8 has joined

  628. COM8 has left

  629. COM8 has joined

  630. COM8 has left

  631. nyco has joined

  632. jabberjocke_ has left

  633. jabberjocke has left

  634. nyco


  635. ralphm bangs gavel

  636. Guus

    I'm just being called

  637. ralphm

    0. Welcome + Agenda

  638. Guus

    I'll lurk for the duration of this call

  639. ralphm

    Hi all!

  640. COM8 has joined

  641. nyco

    so we're 2.5?

  642. nyco

    is it a quorum?

  643. ralphm

    hehe, not yet

  644. ralphm

    Seve and MattJ will surely show up

  645. Guus

    I'm afraid this is going to take a while

  646. Guus

    customer on the phone

  647. jabberjocke_ has joined

  648. jabberjocke has joined

  649. ralphm

    Guus: no worries

  650. nyco

    for a good or a bad news?

  651. COM8 has left

  652. MattJ


  653. nyco


  654. ralphm

    There we go.

  655. Guus


  656. Guus

    count me out guys, sorry

  657. MattJ

    Sorry, I honestly didn't know it was Thursday :)

  658. ralphm

    1. Minute taker

  659. ralphm

    MattJ: happens roughly every 7 days

  660. MattJ

    I'll make a note of that

  661. pdurbin has joined

  662. nyco

    or every 14 days

  663. jabberjocke_ has left

  664. jabberjocke has left

  665. nyco

    I'll do the minutes

  666. ralphm


  667. ralphm

    I thought MattJ was making notes :-D

  668. ralphm

    2. Badges

  669. nyco

    poll finished, sent to board

  670. Alex has left

  671. ralphm


  672. zach has left

  673. ralphm

    So, my summary: overwhelming support for https://opensourcedesign.net/jobs/jobs/2019-03-19-design-of-badges-for-different-xmpp-compliance-levels

  674. zach has joined

  675. Ge0rG

    Is it possible to get the poll results without getting elected into next Board?

  676. ralphm

    And overall enthousiasm for the concept and willingness to use

  677. nyco

    not a problem for me to send those results to members

  678. ralphm

    Ge0rG: sorry for the spoiler

  679. madhur.garg has left

  680. madhur.garg has joined

  681. Ge0rG

    nyco: please do

  682. Ge0rG

    ralphm: I don't mind, as long as we are making progress here

  683. Lance has joined

  684. stpeter has joined

  685. nyco

    we have three tickets regardin badges on the trello board, maybe we can archive two

  686. nyco

    what's next?

  687. ralphm

    I move we choose mray's design, request the assignment of rights, and then publish the respective badges

  688. Nekit has left

  689. ralphm


  690. MattJ


  691. nyco


  692. nyco

    who's in charge?

  693. moparisthebest has joined

  694. Nekit has joined

  695. Ge0rG

    is this about the optical design or also about the content?

  696. ralphm

    The optical design

  697. Ge0rG


  698. ralphm

    I tried to find who was involved, and it seems Ge0rG is

  699. ralphm


  700. ralphm

    (in the 'how to apply' section)

  701. ralphm

    But I can contact mray, if his contact details on this site are valid.

  702. ralphm

    I'll make a new card and archive the others.

  703. Ge0rG

    ralphm: please move on, unless you need some kind of assistance from my side

  704. nyco


  705. ralphm

    Ge0rG: unless you know this person

  706. Ge0rG

    ralphm: not at all. the only contact we had was my posting of the job

  707. pdurbin has left

  708. ralphm

    3. Maxime Buquet => Editor Team

  709. ralphm

    pep. has requested to be on the Editor team

  710. nyco

    that would be cool

  711. ralphm

    Since he's a Member, +1

  712. Seve

    I'm here, my bad guys, was trapped in a meeting

  713. nyco

    what's the procedure here?

  714. ralphm

    we assign him this role, done

  715. nyco


  716. nyco

    —o/ Seve

  717. MattJ


  718. pep.

    And then I request superpowers to the right people?

  719. Seve

    +1 for Maxime Buquet :)

  720. mr.fister has left

  721. nyco

    who can grant that?

  722. mr.fister has joined

  723. ralphm

    Can somebody create a PR for this?

  724. ralphm

    nyco: we do

  725. ralphm

    nyco: Board gets to assign people to work teams

  726. nyco

    I meant superpowers... then I'm interested...

  727. mr.fister has left

  728. mr.fister has joined

  729. MattJ

    pep., get cape, wear cape, fly!

  730. pep.


  731. ralphm

    Ah, I'm sure jonas’ can help out

  732. nyco

    destroy Thanos

  733. ralphm

    or stpeter

  734. lovetox_ has left

  735. ralphm

    (not the destroying, the giving of powers)

  736. nyco


  737. ralphm

    4. Roadmap

  738. ralphm

    I've started this, and some of this has spilled to this room

  739. ralphm

    I've found several areas that need work, in my not so humble opinion, which I should probably fold into a nice e-mail

  740. nyco

    why not a PR?

  741. ralphm

    Examples include: MIX (including how MAM should work), richer messaging (also touching upon MAM in a fairly big way), and establishing Jingle calls in 2019 (with multiple resources, some of which might be not actually connected)

  742. nyco


  743. nyco


  744. Ge0rG

    ralphm: I'm very much interested in a roadmap for the upcoming Compliance Suite 2020, because people have shown interest in the goals we have, and not just the status-quo XEPs

  745. ralphm

    nyco: because I think that a) the list is not complete, b) I'm sure others have opinions

  746. nyco

    I meant: should we add spam to our roadmap?

  747. ralphm

    Ge0rG: the discussion is here on a more general roadmap, not just for the short term, but yes, having an idea of what should be in 2020 is an interesting topic in itself

  748. Seve

    I haven't followed the room lately on this regard, but stating things that need improvement is a grest step forward, would love to see this happen

  749. Nekit has left

  750. Seve


  751. ralphm

    Ge0rG: but I'm not sure how 2020 could contain stuff that isn't mostly already there in some ways

  752. Nekit has joined

  753. stpeter has left

  754. Ge0rG

    ralphm: maybe CS-2020 should have a short-term agenda for 2021, and you prepare the long-term agenda?

  755. Ge0rG

    I don't have any specific ideas yet.

  756. ralphm

    It could. In any case, whatever both lists will end up being, I hope Council will also have opinions on this, before we say: this is it

  757. Seve


  758. zach has left

  759. zach has joined

  760. jonas’

    ralphm, pep., FWIW, someone from iteam with powers on github is needed to grant permissions, I don’t think I have those permissions myself

  761. ralphm

    Ge0rG: so my idea was sending a mail including those examples, and have a bit of discussion, getting to some common ground

  762. ralphm

    jonas’: in that case, MattJ or Kev can help

  763. nyco

    then an email is better than a PR, ok for me

  764. ralphm


  765. Ge0rG

    speaking of iteam, https://mail.jabber.org/pipermail/standards/ is still broken

  766. Ge0rG

    ralphm: yes, please do that

  767. ralphm

    Ge0rG: noted. I'll raise it in their room

  768. Ge0rG


  769. ralphm

    5. AOB

  770. ralphm


  771. ralphm

    Oh, I noticed that JC no longer has time to do the newsletter

  772. Seve

    Yes, wanted to talk with him

  773. Seve

    But basically I would like to discuss with commteam a bit this topic

  774. ralphm

    And has announced that he's no longer doing that

  775. ralphm

    Seve: sure, just a thing to mention

  776. ralphm

    Anything else?

  777. Seve

    I'm part of it, and would like to see first what are we going to do

  778. Seve

    Nothing here

  779. ralphm


  780. ralphm

    6. Date of Next

  781. ralphm


  782. nyco

    I collect links... I wanted to write a bit today, I'll still be in support/help/contrib mode

  783. jmpman has left

  784. ralphm

    7. Close

  785. ralphm

    Thanks all!

  786. ralphm bangs gavel

  787. Ge0rG

    nyco: also please mail the poll results to members@

  788. Seve

    Thank you guys, and sorry for my late show up

  789. nyco

    sir, yes sir

  790. ralphm

    Better late than never

  791. nyco

    no pb

  792. nyco

    thx all

  793. nyco

    I'm on the minutes, then the poll results to members

  794. jonas’

    general remark: I forgot to send out emails for the changes i pushed to xeps on tuesday, I shall do that tonight or at the weekend

  795. ralphm

    jonas’: thanks!

  796. Ge0rG suppresses a sarcastic email about the email archive not being accessible anyway.

  797. Ge0rG suppresses a sarcastic comment about the email archive not being accessible anyway.

  798. jonas’

    well done, Ge0rG!

  799. Mikaela has left

  800. Fahrgast has left

  801. nyco

    both done

  802. j.r has left

  803. j.r has joined

  804. UsL has joined

  805. j.r has left

  806. Nekit has left

  807. Nekit has joined

  808. j.r has joined

  809. Mikaela has joined

  810. Fahrgast has joined

  811. j.r has left

  812. jmpman has joined

  813. adityaborikar has left

  814. adityaborikar has joined

  815. zach has left

  816. zach has joined

  817. COM8 has joined

  818. j.r has joined

  819. COM8 has left

  820. COM8 has joined

  821. COM8 has left

  822. COM8 has joined

  823. COM8 has left

  824. COM8 has joined

  825. COM8 has left

  826. COM8 has joined

  827. zach has left

  828. zach has joined

  829. COM8 has left

  830. COM8 has joined

  831. COM8 has left

  832. adityaborikar has left

  833. adityaborikar has joined

  834. Lance has left

  835. zach has left

  836. zach has joined

  837. j.r has left

  838. stpeter has joined

  839. peter has joined

  840. Nekit has left

  841. Nekit has joined

  842. UsL has left

  843. Lance has joined

  844. peter has left

  845. COM8 has joined

  846. Nekit has left

  847. Nekit has joined

  848. zach has left

  849. zach has joined

  850. COM8 has left

  851. COM8 has joined

  852. ralphm


  853. COM8 has left

  854. jubalh has left

  855. Fahrgast has left

  856. UsL has joined

  857. sezuan has left

  858. Fahrgast has joined

  859. COM8 has joined

  860. stpeter has left

  861. pdurbin has joined

  862. jmpman has left

  863. COM8 has left

  864. j.r has joined

  865. waqas has joined

  866. zach has left

  867. zach has joined

  868. pdurbin has left

  869. mr.fister has left

  870. Mikaela has left

  871. Mikaela has joined

  872. jmpman has joined

  873. COM8 has joined

  874. waqas has left

  875. COM8 has left

  876. UsL has left

  877. zach has left

  878. zach has joined

  879. morgan has joined

  880. morgan has left

  881. Steve Kille has left

  882. Wojtek has joined

  883. Wojtek has left

  884. morgan has joined

  885. Steve Kille has joined

  886. morgan has left

  887. Nekit has left

  888. Nekit has joined

  889. zach has left

  890. zach has joined

  891. Fahrgast has left

  892. Chobbes has joined

  893. valo has left

  894. valo has joined

  895. morgan has joined

  896. Chobbes has left

  897. morgan has left

  898. waqas has joined

  899. Chobbes has joined

  900. Fahrgast has joined

  901. moparisthebest has left

  902. jmpman has left

  903. LNJ has left

  904. moparisthebest has joined

  905. morgan has joined

  906. zach has left

  907. zach has joined

  908. jubalh has joined

  909. LNJ has joined

  910. jubalh has left

  911. lovetox has joined

  912. morgan has left

  913. waqas has left

  914. Alex has joined

  915. zach has left

  916. zach has joined

  917. mr.fister has joined

  918. Fahrgast has left

  919. Fahrgast has joined

  920. Nekit has left

  921. morgan has joined

  922. Chobbes has left

  923. Steve Kille has left

  924. zach has left

  925. zach has joined

  926. peter has joined

  927. stpeter has joined

  928. jubalh has joined

  929. Fahrgast has left

  930. peter has left

  931. j.r has left

  932. pdurbin has joined

  933. Fahrgast has joined

  934. morgan has left

  935. zach has left

  936. pdurbin has left

  937. zach has joined

  938. waqas has joined

  939. debacle has joined

  940. stpeter has left

  941. adityaborikar has left

  942. waqas has left

  943. waqas has joined

  944. lovetox has left

  945. zach has left

  946. zach has joined

  947. j.r has joined

  948. lovetox has joined

  949. morgan has joined

  950. morgan has left

  951. morgan has joined

  952. zach has left

  953. zach has joined

  954. stpeter has joined

  955. peter has joined

  956. Fahrgast has left

  957. peter has left

  958. morgan has left

  959. morgan has joined

  960. jabberjocke_ has joined

  961. jabberjocke has joined

  962. debacle has left

  963. Fahrgast has joined

  964. morgan has left

  965. morgan has joined

  966. zach has left

  967. zach has joined

  968. jubalh has left

  969. stpeter has left

  970. sezuan has joined

  971. peter has joined

  972. stpeter has joined

  973. peter has left

  974. jubalh has joined

  975. zach has left

  976. zach has joined

  977. jubalh

    > I'm living between Munich and Stuttgart (South Germany) Not far from me

  978. Douglas Terabyte has left

  979. stpeter has left

  980. eevvoor has joined

  981. Douglas Terabyte has joined

  982. eevvoor has left

  983. Douglas Terabyte has left

  984. Douglas Terabyte has joined

  985. zach has left

  986. zach has joined

  987. LNJ has left

  988. pdurbin has joined

  989. morgan has left

  990. morgan has joined

  991. LNJ has joined

  992. zach has left

  993. zach has joined

  994. lumi has joined

  995. Douglas Terabyte has left

  996. Fahrgast has left

  997. sezuan has left

  998. pdurbin has left

  999. Douglas Terabyte has joined

  1000. UsL has joined

  1001. Nekit has joined

  1002. sezuan has joined

  1003. Yagiza has left

  1004. Yagiza has joined

  1005. j.r has left

  1006. Lance has left

  1007. j.r has joined

  1008. zach has left

  1009. zach has joined

  1010. UsL has left

  1011. reda-alaoui has joined

  1012. sezuan has left

  1013. kokonoe has left

  1014. Fahrgast has joined

  1015. reda-alaoui

    Hi everyone, I am implementing XEP-0313 Message Archive Management on the server side. Suppose the server supports offline storage (in addition to MAM) and the recipient of the sent message is offline. A message is sent to the offline recipient. Should the message be archived in the recipient archive WHEN: - the message is received by the recipient's server OR - the recipient turns back online

  1016. kokonoe has joined

  1017. Zash

    The former I think

  1018. Zash

    Interaction between message archive and offline message store is something that I don't think is fully figured out yet

  1019. reda-alaoui

    Thank you Zash , but that means that when the recipient client comes back online, it will probably pull the recipient archive containing the archived message while the server will be pushing the offline stored message. How will the client be able to know that the pushed message is the same as the archived message?

  1020. zach has left

  1021. zach has joined

  1022. Zash


  1023. Alex has left

  1024. reda-alaoui

    Ok I thought stanza-id were limited to XEP-0313. From my understanding stanza-id is set by the server. How can we provide the stanza-id outside of XEP-0313 IQ queries?

  1025. Fahrgast has left

  1026. Zash

    The way it goes in Prosody is: - Message comes in - Message is added to the archive - The archive ID is attached to the stanza - Delivery is attempted - If there were no online clients with non-negative priority to accept the stanza, add to offline store (with the stanza-id)

  1027. Zash

    Not optimal, but it is the way it is because of historical reasons and the thing being modular.

  1028. reda-alaoui

    So you are adding stanza-id to the offline storage mechanism even if it is not described in https://xmpp.org/extensions/xep-0160.html? Or am I missing a fundamental specification that introduced stanza-id for every message?

  1029. Yagiza has left

  1030. Zash

    The module that handles MAM adds it when it adds the stanza to the archive. Then it just tags along through the rest of the chain of events.

  1031. morgan has left

  1032. Ge0rG

    Zash: when do Carbons happen?

  1033. Zash

    Ge0rG: Somewhere in the middle

  1034. Zash

    Before normal delivery I think

  1035. Ge0rG

    reda-alaoui: ideally, you could implement offline storage as a pointer into the MAM archive. When a client fetches from MAM, that pointer is moved forward. Under ideal conditions, you end up with the pointer after the end of the archive and no need to send anything after the MAM sync

  1036. lovetox

    reda-alaoui, XEP313 says you MUST add the stanza-id to every message that is archived

  1037. Ge0rG

    There are some corner cases with MAM limited to roster or clients only fetching a subset

  1038. pep.

    So.. we're stuck on pretty basic stuff on slix while trying to add async functionalities in core parts. How do people even guarantee in-order delivery of stanzas? :x

  1039. lovetox

    so if you send a message that is in your offline store AND mam archive, but dont attach a stanza-id to the offline message

  1040. lovetox

    you are violating 313

  1041. Ge0rG

    pep.: you can't with MAM

  1042. lovetox

    because now you sent a message without stanza-id that is in fact in the MAM archive

  1043. reda-alaoui

    lovetox yes I know but I didn't know it should be propagated to other specs

  1044. pep.

    Ge0rG, not talking about MAM

  1045. mathieui

    Ge0rG, we’re more in an omemo case

  1046. Ge0rG

    lovetox: technically, you could have a separate store for offline that's filled before the message gets a MAM ID

  1047. pep.

    But it's not really omemo related either tbh

  1048. reda-alaoui

    Ge0rG : that's what we have currently

  1049. Ge0rG

    mathieui: stupid specs come with stupid consequences for the implementation

  1050. lovetox

    Ge0rG, it does not matter really, because in the moment you send a message from the offline store, you probably would put it into mam also

  1051. lovetox

    and then you have to inject a stanza-id anyway

  1052. Ge0rG

    reda-alaoui: but it helps a client tremendously to deduplicate, if the MAM ID is always sent

  1053. lovetox

    Ge0rG, its not about help

  1054. lovetox

    its mandatory

  1055. reda-alaoui

    Ge0rG yes of course, it is a must have

  1056. reda-alaoui

    ok, thanks a lot !

  1057. reda-alaoui

    back to my code then :p

  1058. remko has left

  1059. lovetox

    thing is a client cant differentiate between a offline message

  1060. lovetox

    and a live message

  1061. lovetox

    this means i receive suddenly messages from my mam:2 server that have no stanza-id, this would in best case drop a warning

  1062. lovetox

    actually just lets not test this :)

  1063. Zash

    Implementation details galore

  1064. lovetox

    worst case is, i have to fallback to timestamp/body based deduplication

  1065. lovetox

    or even worse a client doesnt implement that because he only supports mam:2 and depends on dedup with stanza.id

  1066. Zash

    Transition periods are messy

  1067. lovetox

    i have to implement that offline message purge

  1068. Ge0rG

    lovetox: the paragraph you quoted can be read as "the server must add the ID to the archived message only"

  1069. Daniel

    pep.: my outgoing stanza queue is synchronized

  1070. lovetox

    yeah yeah, i know we can read everything the other way, you are right its a grey area that was not thought about when spec mam

  1071. lovetox

    so yeah please server developer add stanza-id :D

  1072. Daniel

    So if I need stanzas in order I write them to that queue from the same thread and everything is fine

  1073. pep.

    Daniel, how do you handle OMEMO encryption for exapmle, where you have to fetch devicelists and bundles and only then send your encrypted message

  1074. Zash

    Would it be sane that if you set MAM defalut to roster, to then reject anything that doesn't go in the archive when you're offline?

  1075. Zash

    As in, bounce with an error

  1076. Daniel

    pep.: call backs

  1077. Ge0rG

    > servers SHOULD include the element as a child of the forwarded message when using Message Carbons (XEP-0280) Another non-requirement.

  1078. pep.

    Daniel, Ok so you block sending on the stream?

  1079. zach has left

  1080. pep.

    Well, yeah it's synchronized

  1081. Steve Kille has joined

  1082. zach has joined

  1083. Fahrgast has joined

  1084. Zash

    Ge0rG: Make it a MUST for routing 2.0 or whatsitcalled

  1085. LNJ has left

  1086. Daniel

    No I guess you could send other stanzas while waiting for the omemo fetches to return

  1087. Ge0rG

    Zash: will you implement that in prosody?

  1088. pep.

    Daniel, and then you need to keep track of what came in what order, but you also need to prioritize the OMEMO IQs to be able to send that one message

  1089. Zash

    Ge0rG: I make no promises about what I do with my free time.

  1090. Ge0rG

    I've just realized that smack will process messages asynchronously, so when I page through 2K of MAM messages, and update the progress counter accordingly, it will arrive at 100% while the async handler is still busy somewhere in the first half

  1091. lovetox

    update it on receiving the end of the page?

  1092. Daniel

    pep.: well I guess I can ensure order for a particular set of stanza but I don't have to. For example I can make sure that a leave and a join presence are always send after each other

  1093. Daniel

    Also the omemo message stanza waiting to be encrypted is not in the queue

  1094. Daniel


  1095. pep.

    In poezio atm, I am implementing the encrypt method as a stream filter, i.e., I want to catch everything going out to be sure none of what should be encrypted is sent in plain

  1096. j.r has left

  1097. Daniel

    I send plain text into the omemo pipeline and only after it was encrypted it becomes a stanza and gets fed into the queue

  1098. j.r has joined

  1099. pep.

    That means I can just send a normal message, and my filter will take care of the encryption

  1100. pep.

    But that blocks the whole loop

  1101. Daniel

    > In poezio atm, I am implementing the encrypt method as a stream filter, i.e., I want to catch everything going out to be sure none of what should be encrypted is sent in plain Yeah I'm not a fan of that approach

  1102. Daniel

    That's what caused the gajim otr html body leak

  1103. pep.


  1104. lovetox

    pep. on hitting send, i check if every pre condition is met for encrypting

  1105. pep.

    The point is to prevent that

  1106. lovetox

    and if not, just nothing happens

  1107. lovetox

    text is still in input

  1108. pep.

    Otherwise I have no guarantee poezio won't send plain stuff behind my back

  1109. UsL has joined

  1110. lovetox

    i query whatever needs to be queried

  1111. pep.

    (chatstates, etc.)

  1112. pep.

    (and other plugins)

  1113. Fahrgast has left

  1114. Daniel

    pep.: but if I understand you correctly what you are doing is creating a message stanza with a plain text body and then your filter removes the body and inserts omemo. I never do that. An omemo message for me is never a message stanza

  1115. Daniel

    pep.: but if I understand you correctly what you are doing is creating a message stanza with a plain text body and then your filter removes the body and inserts omemo. I never do that. An omemo message for me is never a message stanza with a normal body

  1116. lovetox

    Daniel because you dont have a plugin system

  1117. pep.


  1118. Daniel

    Think of stanzas as being immutable.

  1119. pep.

    Daniel, we're in python

  1120. pep.

    Nothing is immutable

  1121. Daniel

    There are not really in my code but I treat them as such

  1122. Daniel

    You can still think of them as such

  1123. Daniel

    lovetox: yeah I don't like that kind of plugin systems

  1124. pep.

    plugin system or not, I have to review every bits of poezio to ensure I won't leak plain. Stream filters were kind of the escape hatch that would allow me not to do that

  1125. jcbrand has left

  1126. pep.

    plugin system or not, I have to review every bits of poezio to ensure I won't leak plain. Stream filters were kind of the escape hatch that would have allowed me not to do that

  1127. lovetox

    Daniel what you do you can only do if the client has knowledge about encryption

  1128. Daniel

    lovetox: yes. Obviously

  1129. mathieui

    the basic use case ("press enter") is fine, the issue is the rest of the commands and such (/correct, etc) which we have to check and disable

  1130. pep.

    Also that means it doesn't make sense to implement OMEMO as a plugin in poezio anymore, without stream filters

  1131. lovetox

    yeah, Gajim has no knowledge about encryption, it just passes the stanza right before sending to the encryption plugin

  1132. pep.

    And that's not possible license-wise atm

  1133. Nekit has left

  1134. lovetox

    mathieui, correct is perfectly fine to use with omemo

  1135. mathieui

    lovetox, yes, but the point is we need to not forget any command which might send plaintext

  1136. lovetox

    yeah thats a really error prone way of doing this i think

  1137. lovetox

    instead poezio should not care at all about it

  1138. pep.

    Which is why I went the stream filter way.. :P

  1139. lovetox

    do everything with the stanza as it would normally

  1140. lovetox

    and at the end omemo runs it through a whitelist of stanza nodes

  1141. mathieui

    lovetox, that was the plan

  1142. Daniel

    Is there an otr plugin for poezio?

  1143. pep.


  1144. pep.

    Which is..

  1145. Daniel

    Because for that you also have to wait

  1146. Daniel

    Even for remote peers. Which is worse

  1147. pep.

    I actually have no clue, I won't pronounce myself on it :x

  1148. lovetox

    but this has nothing to do with the problem pep. described i think

  1149. mathieui

    Daniel, the OTR plugin is implemented in the way I described above

  1150. lovetox

    you should have a handler that checks before creating a stanza, even before removing the message from the input field

  1151. lovetox

    if encryption is even possible

  1152. mathieui

    which is *really* ugly (whitelist elements, intercept the code that sends messages etc)

  1153. lovetox

    not pass it to haf of poezio than at the end find out we cant even encrypt because key is missing

  1154. pep.

    lovetox, "before creating a stanza" doesn't map as you think it does, in slix

  1155. lovetox

    this has nothing to do with slix

  1156. lovetox

    that is UI code

  1157. lovetox

    you press the button

  1158. pep.


  1159. lovetox

    on button pres, you check

  1160. Tobias has left

  1161. Daniel

    > lovetox, "before creating a stanza" doesn't map as you think it does, in slix Then do it in poezio. Not in slix

  1162. pep.

    nowhere I mentioned UI :P

  1163. pep.

    I actually don't care about UI for now

  1164. Ge0rG

    This is the moment when you introduce numeric stanza filter priorities, so that you can do encryption last.

  1165. pep.

    Daniel, yeah, slix/poezio ~

  1166. Zash

    And then you need something *really last*

  1167. Ge0rG

    Zash: thus numeric.

  1168. lovetox

    pep., you check before the message would vanish from the input

  1169. Zash


  1170. Ge0rG

    And then a plugin author thinks they must be laster than last and change the body post encryption.

  1171. Daniel

    Did I mention I don't like plug-ins?

  1172. Zash

    Double-edged sword if anything

  1173. lovetox

    also writing a client is one thing

  1174. lovetox

    writing a client that perfectly interfaces with all kind of plugins

  1175. lovetox

    a whole other game :)

  1176. Ge0rG

    Just write a plugin management system and let the modules do the rest.

  1177. kokonoe has left

  1178. Ge0rG

    If you are okay with Lua, there is a decent plugin management system that's handling xml streams already.

  1179. Zash

    Ge0rG: You know how Prosody is actually a Lua variant of node.js with some plugin management on top?

  1180. Ge0rG

    Zash: no.

  1181. Zash

    Unheard of!

  1182. Ge0rG

    Zash: prosody requires a VirtualHost, so it can't be what you claim it is.

  1183. pep.

    Daniel, yeah that's a different discussion, I don't have a choice anymore :)

  1184. kokonoe has joined

  1185. Fahrgast has joined

  1186. Daniel

    In any case I suppose you currently have something of a `on_message` hook that has a message stanza as a parameter. What you probably want to do is create another hook `on_message_submitted` that has the plain text of the input and gets fired as soon as the user presses enter.

  1187. zach has left

  1188. zach has joined

  1189. Daniel

    And if that hook returns true it gets into the normal pipeline and the input field gets cleared

  1190. Daniel

    And for omemo message you handle it differently

  1191. Ge0rG

    But then you need an internal representation for all message types, which is then mapped to the respective xml on transmission.

  1192. Daniel

    And then after omemo ran over it you can re-emit the message stanza and run it though the chain

  1193. pep.

    Daniel, yeah, it's already what I'm doing but at the slix level. I'd need to look into it because atm it seems to me I'm going to run into the same problems

  1194. Daniel

    Same problem = leaking stuff you don't want to leak?

  1195. pep.


  1196. pep.

    I lose in-order guarantees

  1197. pep.

    Either that or I block my stream for ages while fetching devicelists and bundles

  1198. pep.

    https://lab.louiz.org/poezio/slixmpp/merge_requests/24/diffs that's supposed to help here, but it's still not what we want

  1199. peter has joined

  1200. peter has left

  1201. Chobbes has joined

  1202. alameyo has left

  1203. alameyo has joined

  1204. Alex has joined

  1205. jubalh has left

  1206. Lance has joined

  1207. madhur.garg has left

  1208. madhur.garg has joined

  1209. wurstsalat has left

  1210. Chobbes has left

  1211. Chobbes has joined

  1212. zach has left

  1213. zach has joined

  1214. Fahrgast has left

  1215. Chobbes has left

  1216. Chobbes has joined

  1217. UsL has left

  1218. Fahrgast has joined

  1219. stpeter has joined

  1220. peter has joined

  1221. karoshi has left

  1222. kokonoe has left

  1223. kokonoe has joined

  1224. lovetox has left

  1225. Steve Kille has left

  1226. Chobbes has left

  1227. goffi has left

  1228. Mikaela has left

  1229. zach has left

  1230. zach has joined

  1231. pdurbin has joined

  1232. lskdjf has left

  1233. larma has left

  1234. kokonoe has left

  1235. kokonoe has joined

  1236. reda-alaoui has left

  1237. pdurbin has left

  1238. Allo has left

  1239. Allo has joined

  1240. peter has left

  1241. lumi has left

  1242. Zash has left