XSF Discussion - 2019-08-22

what has time to do with that?

Yes you need to start mam or get the last ID *before* you send presence ✏

Don't think carbons have anything to do with that

you could enable carbons before sending presence, too

but don’t you need to to MAM *after* sending presence/enabling carbons to not lose a message?

otherwise, if you got the last ID / queried MAM *before* sending presence, a message might’ve arrived between querying the ID and sending presence/enabling carbons, which you’d then miss

Mhhh ideally then you'd get the last ID before presence but fire the query after

sure

no, wit

*wait

if you get the last ID before presence, what happens if a message arrives between you getting the last ID and you sending presence?

jonas’: last ID defines the lower end

what does "lower" mean in that context?

Upper end is undefined

is last ID the last thing you saw in your last session?

or is last ID the newest ID?

Yes

The newest you currently have locally

But you won't get newer messages until you sent presence.

Daniel, I see!

that makes sense then

it’s early in the morning, sorry :)

Daniel: if you're saying there's a specific order of events that ensures you get all messages, is that documented somewhere?

Like: get last id, send carbons request, send presence, request mam?

Because XEP-0313 says client sync is it of scope (!), and refers to bind2.

out of scope

ralphm: an attempt was made to document it in 0313, but it turned out not to work.

what didn’t work?

the attempt?

Daniel: the actual ordering that was written down, I think

Ge0rG: so is there an order that does work? Or can it not be solved with current protocols?

the order that ralphm just wrote down doesn’t work?

So you note the last id you know, send presence, and then fetch MAM in parallel to receiving offline messages / live traffic? Is there any change to sync linearly in the actual message order?

I believe that the order I mentioned is at-least-once. There's no such thing as exactly-once in general.

well receiving MAM and offline messages in parallel isn’t that bad; you can dedup them

Right, there's no way to avoid dedup.

however when i detect that MAM preferences are set to always and server has offline msg mangament things i also run a purge

before sending presence

… which can reduce the number of messages to dedup, yes.

but i see that only as a minor traffic optimization

You also need to dedup between the Carbons of live messages that arrived between you enabling Carbons and you sending presence, and their respective offline copy

Daniel: is the filter for messages going to MAM and offline identical?

ralphm: probably not

my dedup would work fine; and does on servers that don’t have the purge thing

ralphm, i guess there is nothing in the protocol to ensure that

Ge0rG: you don't get carbons until you send presence

Daniel: if you purge offline, isn't there a chance that you'll purge too much?

ralphm: are you sure?

Ge0rG, well my reasoning is if it doesn’t end up in MAM it wasn’t important

your second client wouldn’t receive it either

That's the idea. Before sending presence, you only get message stanzas directed at your full JID.

i mean if the rules between offline messages and mam differ it is probably time to reconsider the rules

ralphm: I don't see presence mentioned in 0280: > When the server receives a <message/> eligible for carbons delivery addressed to a client JID (either bare or full), it delivers the <message/> according to RFC 6121 § 8.5.3, and then delivers a forwarded copy to each Carbons-enabled resource for the matching bare JID recipient that did not receive it under the RFC 6121 delivery rules.

but if you hate the purge don’t purge

that's not something i would codify anywhere

especially since purge capability isn’t really widespread anywhere

Ge0rG: hmm, that surprises me. Curious how servers implement this then.

I’m pretty sure that some people use carbons-enabled clients with negative priority to get all messages (also) on a non-carbons-enabled client :)

Daniel: I remember complaining about a race condition there, but I can't say right now what it was

jonas’: I'm doing that, but a negative presence still is a presence

Right

Just for my information, though, the only upshot of that combination (negative presence and carbons) is that you don't get message stanzas to bare JID that don't match the carbon criteria?

I'm curious in what situation that is beneficial.

ralphm, the non-carbons-enabled client will get all the messages because it’s the only one with >= 0 priority

while the carbons-enabled clients also get the messages because they’ve enabled carbons

But only of type chat, for example.

So just a trick to get carbons behavior with n clients although only n-1 clients support carbons.

"good enough"

Hmm

not that implementing carbons in a client is hard…

It is interesting though. Most clients use presence priority to figure out which 'show' to display.

So in this setting that might mislead

Obscure edge case, maybe

Yes.

Any way, I think that servers will still wait for presence to start sending any messages.

(that were not to this connection's resource)

It does not matter in what order you activate carbons, presence, offline messages

you can never miss a message if you do MAM last

thats the easy way, that might have here and there some message that has to be deduped

but thats no problem with stanza.-id

lovetox_: what was said is that before sending all that, you need to establish the last id you have.

The rest of the order is less relevant, although there might be an optimum to minimize traffic of dupes.

you have the last ID in your database

Yes, you just need to make sure you don't invalidate it by getting new messages before doing MAM

yes of course, before MAM catchup is not complete you should not record "last stanza id"

and MAM catchup will always give you the very last message

That was the thing we tried to convey

I'm still curious as to why it is said to be out of scope and mentions bind2 as possible remedy.

So is it guaranteed that servers don't mix in live messages while sending MAM messages?

flow: I don't think so

That seems complex to implement

ralphm, I think the idea was to have a single sync point where you get the ID of the most recent message in your MAM (or possible multiple IDs based on the sender JID) and enable carbons and message delivery in a single server-atomic operation

removing the need to deduplicate there

Yes, essentially you need an atomic operation where you enable live traffic and receive the ID of the last non live message

So that you can fill the hole.

But there is no way to actually receive what you need in a linear order.

true

and there is no way to receive the messages exactly once.

flow why would it make a difference if a server mixes in live messages?

Ge0rG, with bind2, there would be, or?

jonas’: maybe

"maybe"?

jonas’: if bind2 also removes all MAMed messages from the offline history sync

in my world, offline messages do not exist and you always purge them.

jonas’: that sync point seems to be when the server receives presence

ralphm, it’s not a sync point, because you don’t get the "newest MAM ID" from that

(you = the client)

It is a sync point for live messages

how is it sync if it’s one-way?

Well, sure you still need to do the mam query afterwards. I don't see how one can technically achieve exactly once.

ralphm, if the server replied (to the presence, or to whatever) with the ID of the most-recent stanza in the MAM archive at the time the presence was received

then you’d query from "last stanza ID I’ve seen" until "most-recent stanza ID I got from presence"

you dont record that jonas

lovetox_, ?

you dont record last-stanza-id until you made your mam query

and while the mam query is running you ignore all stanza-ids from live and carbons

From a client perspective they'd be ideal. I don't think that's easy to implement server side. Also, exactly once delivery is theoretically impossible.

you only look at the mam query

ralphm, sure.

the last message in the mam query, is your most recent message, then you record the stanza-id

and then you are fully synced

and it does not matter what the server sends you inbetween your mam query

if you don’t want to be efficient, sure.

I've also seen MAM implementations that were eventually consistent, by the way.

yes jonas thats what we were saying

(server side)

and nobody needs to be efficient here, mam is complex enough

a few duplicates are not bad, its totally fine

and you can now overengineer that so there will never be a message twice, but its not an "issue" of clients that we are so inefficient because we get 3 offline messages while mam query

and btw you have the same thing with MUC

live messages will arrive while you query mam

Oh, don't get me started on MUC or MIX.

i would argue this was a problem back in mam:0 mam:1 days

if you could not depend on stanza-id

now with mam:2, really i dont care about a handful of duplicates in a mam catchup

Sure.

Problem with group chats is that you have to do it for each room.

And although MIX currently says you should also store in your local archive (which I like, as it decreases client complexity), it is unclear how a server catches up to the room archive in case of delivery issues (e.g. s2s outages).

ralphm what is the problem with making a query to each MUC? inefficient?

yes maybe i can spare 20 stanzas, for 20 muc joins, but other than that, this just works

if i switch now to some other concept, like you just decribe and for that get real problems like s2s catchup etc

i would stay with the old behaviour

Oh I didn't know that the server was supposed to catch up on your behalf. I thought you were just going to miss messages

Or magically know where the gaps are

lovetox_: on mobile clients, it makes sense to reduce: round-trips, wakeups, data bandwidth

yes, i have to trust you on that

though i would say mobile clients with stream management should not run to often into the, total connection reset scenario

Ge0rG, well you'd do mam and join at the same time. so it's not more roundtrips or more wake ups

all it does it redcue traffic

which is fair

Daniel: well, let's say it is unclear. The reason I dislike having to check each channel for its archives, is that it doesn't scale. If you want to implement something like Slack, you can easily be in 40+ channels. In WhatsApp, my daughter is in uncountable ones that are never left.

ralphm: yes, we need a way to start a session by just telling the server "my last local ID is X, give me everything relevant after that"

ralphm, depends on what you want to scale and how it is implemented server side. send a message to a group of 500 people - you might not want to store it 501 times in different MAM archives

especially not in 501 transactions

Daniel: I strongly believe that the collective archive of all my conversations should be as close to me as possible. I.e. my own server.

I'm sure there are server side optimizations that could be done.

Daniel: having 501 archives wasn't a problem for the designers of MIX

Ge0rG, yes we are talking about mix

(or muc/sub which does similar things for that matter)

I mean we are also sending 501 copies over the wire.

ralphm, i get that. i'm with you on that. however if we don’t change how MAM is currently implemented it will trigger 501 transactions

Daniel: I think the only actual efficiency problem about MIX is that you end up with 501 copies of each message, with 490 accounts being abandoned by their owner

Even if participants are on the same server. And not counting c2s copies

which is actually a problem we are running into on one project

Daniel: a server implementing both the MIX and the account domain could use the same archive table, and just remember for each account when it last joined

If the problem is retention, then there are two ways: limit in time/space, or pay for storage.

speaking of MAM, where are MUC-PMs stored?

for us it was not necessary the storage but the throughput

501 transactions are expensive

Daniel: did you see my 'eventually consistent' remark above?

ralphm, your point being we just have to do it anyway?

Ge0rG on your account MAM

if this was the question

lovetox_: yes

only type=groupchat is in the MUC archive, usually on current servers

Daniel: I'd prefer to have one, complete, archive per user, and only use room archives to catch up with pre-join history, but I'm not sure if this is possible. It would surely involve backfilling, and I think that's at the very least undefined, and impossible at worst, with the way MAM is designed right now.

yes you get your PMs from your normal catchup

ralphm, yes to all that

and also rather expensive with the way MAM is designed right now

Availability of room history is deemed critical enough to dive into the headaches of maintaining local copies even though availibility of the room service itself isn't considered critical enough to distribute it across domains?

did someone say matrix?

So we get the cons of Matrix without the pros?

Hah.

Not sure we need to agree on a model though. I guess local MIX archives can remain an optional feature without introducing trouble?

as long as it is advertised and clients know what to do

Holger just out of interest how many messages does the conversations.im server MAM archive contain currently ?

alot.jpg

i think you have no MAM storage time limit on that server if i remember correctly

we do; because we ran out of disk space :-)

haha 😃

This came totally unexpected.

Holger: sure, Matrix solves some these issues, but I believe they haven't tackled group scalability fully yet, and if we're talking about possibly having dimensions in MAM archives for reactions, edits, and other attachments to messages, I am not sure it handles that well at all.

Having just a dag of commits is not great in that respect

couldn't an account server "cheat" by just forwarding all MAM queries to the MIX service?

Ge0rG, only if you were to filter by the MIX service

my understanding is that a normal MAM query would give you MIX and non-MIX messages interleaved

Ge0rG: that might not complete in a reasonable time, and still leave you with gaps?

What jonas’ says

also how would your server know the relevant stanza-id of the mix service

Eww.

Right. So many problems.

As I said, this is not easy

By the way, somebody remarked it being ok to send MAM along with join, but join is a one time thing in MIX. After that your server will just send presence.

that remark was about MUC

Ok

If only we had threads.

😭

is this the "Can of Worms" day?

So to sum it up: Carbons and MAM are ugly workarounds to make an unreliable message "delivery" protocol more reliable, and MIX is codifying those workarounds without solving the actual reliability problems?

jonas’: like every day.

Ge0rG: maybe.

So I need to have a flag for type=groupchat messages, to exclude those from the last-id-query, while retaining other-typed messages from MUC JIDs, like invitation and invitation rejection.

yes

And I need a DB index on that flag.

or do the gajim where you store the last mam ids seperatly

What could possibly go wrong with that?

probably the same amount of things that can go wrong with your approach

only different things

I'm still wondering, given a set of IDs, how do I determine which one is last?

The anticipated answer is: I can't, I need to know from the context

the gajim way(tm) also gives you the ability to remove messages locally while still making the catchup work

and not re-fetch a deleted message

there are forked versions of Conversations that do the gajim for that purpose

Yeah, that makes some sense.

Ge0rG i would suggest to hold the last stanza id per jid, in a separate table

and yes stanza-id has no order

lovetox_: is there anything else one should put into that table?

so just from the id you cant determine if its the highest or lowest

its opaque

lovetox_: you update that table when you finish a MAM query?

no on every MAM message

so if the catchup breaks up or my client crashes

lovetox_: on every in-order MAM message? Or are all your MAM messages in-order?

i dont have to refetch

Sigh. I wish MAM would go away and server would just track the last delivered message per client resource.

it would be so easy with 0198, at least for the sake of full-sync clients

i contact you later Ge0rG with the db schema and what i store, must go now

lovetox_: thanks!

lovetox_> flow why would it make a difference if a server mixes in live messages? I guess it depends if a live message triggers something that touches the processing of the incoming MAM archive messages. I was just wondering what people think about that

For me Gajim never seems to properly retrieve / show MUC history after coming online.

What's the status of the Badges poll, BTW? I'd _really_ like to know the results.

ralphm this works. but it only shows what you missed

so if you join twice, all history messages are in the history window CTRL + H

what is a history window and why should it even be there?

a windows where you can browse history ✏

search filter etc

I never understood that. I want the history in the respective chat tab, with a dynamic filter if possible

i want that too but i have not time to do it 🙂

as you may know Gajim is 15+ years

Yeah.

and dynamic history loading in the same window was not a thing 10 years back

Also people weirdly love that

which reminds me. Tomorrow is yaxim's 10th birthday.

Daniel: people love the history window?

Yes

Weird indeed

You just know weird people.

most people are weird

also displaying only in the chat window is not a solution for all circumstances

lovetox_: now I'm curious

i may want to look at history with contacts that are not in my roster anymore

or MUCs i dont want to join anymore

having it *only* in the chat has drawbacks

lovetox_: what's wrong with displaying those in the respective chat tabs, without joining?

this means i have to have a concept of all mucs i ever joined, must be able to open them and look at history *without* joining

and how do i know if the user wants to only look at history or want to join=?

etc etc

Yes.

this is of course the easy thing for 90% of all use cases

And space bar heating

Yesterday I ran hashcat on my laptop's Intel GPU. There was a frightening smell of something burnt. And space bar heating.

Ge0rG : what i save in the table is, archive_jid, last_mam_id, last_timestamp

i save the timestamp because i ran into the following problem

client is turned off 2 weeks, i join a muc and i need to know that the last mam id is 2 weeks back, because i dont want to request that many messages

so i can now switch to a timebased query, like give me last 2 days

i apply this only to public mucs

interesting, thanks

lovetox_: is archive_jid the JID of the MAM service? Do you have your own domain in there for account MAM?

yes

no

its just the jid of the archive

and yes my own jid is in this table

but i apply different rules to account mam query, i always request everything there, except on very first start

what do you request on first start?

i think 7 days

but i have a button in gajim, that lets you sync any timeperiod before

into the history window? ;)

yes

this is not a button per chat

this is a account button, sync everything, last 6 months, last 3 months

like one time shot, after setting up a client, do i want to have my whole archive locally or not

users sometimes want to test something and set up a new instance etc, in this case i dont want to download 10.000 messages

time?

I'm just being called

0. Welcome + Agenda

I'll lurk for the duration of this call

Hi all!

so we're 2.5?

is it a quorum?

hehe, not yet

Seve and MattJ will surely show up

I'm afraid this is going to take a while

customer on the phone

Guus: no worries

for a good or a bad news?

Hey

cool

There we go.

unsure

count me out guys, sorry

Sorry, I honestly didn't know it was Thursday :)

1. Minute taker

MattJ: happens roughly every 7 days

I'll make a note of that

or every 14 days

I'll do the minutes

Thanks

I thought MattJ was making notes :-D

2. Badges

poll finished, sent to board

Right

So, my summary: overwhelming support for https://opensourcedesign.net/jobs/jobs/2019-03-19-design-of-badges-for-different-xmpp-compliance-levels

Is it possible to get the poll results without getting elected into next Board?

And overall enthousiasm for the concept and willingness to use

not a problem for me to send those results to members

Ge0rG: sorry for the spoiler

nyco: please do

ralphm: I don't mind, as long as we are making progress here

we have three tickets regardin badges on the trello board, maybe we can archive two

what's next?

I move we choose mray's design, request the assignment of rights, and then publish the respective badges

+1

+1

+1

who's in charge?

is this about the optical design or also about the content?

The optical design

yay!

I tried to find who was involved, and it seems Ge0rG is

:-D

(in the 'how to apply' section)

But I can contact mray, if his contact details on this site are valid.

I'll make a new card and archive the others.

ralphm: please move on, unless you need some kind of assistance from my side

thx

Ge0rG: unless you know this person

ralphm: not at all. the only contact we had was my posting of the job

3. Maxime Buquet => Editor Team

pep. has requested to be on the Editor team

that would be cool

Since he's a Member, +1

I'm here, my bad guys, was trapped in a meeting

what's the procedure here?

we assign him this role, done

+1

—o/ Seve

+1

And then I request superpowers to the right people?

+1 for Maxime Buquet :)

who can grant that?

Can somebody create a PR for this?

nyco: we do

nyco: Board gets to assign people to work teams

I meant superpowers... then I'm interested...

pep., get cape, wear cape, fly!

o/

Ah, I'm sure jonas’ can help out

destroy Thanos

or stpeter

(not the destroying, the giving of powers)

:'(

4. Roadmap

I've started this, and some of this has spilled to this room

I've found several areas that need work, in my not so humble opinion, which I should probably fold into a nice e-mail

why not a PR?

Examples include: MIX (including how MAM should work), richer messaging (also touching upon MAM in a fairly big way), and establishing Jingle calls in 2019 (with multiple resources, some of which might be not actually connected)

good

spam?

ralphm: I'm very much interested in a roadmap for the upcoming Compliance Suite 2020, because people have shown interest in the goals we have, and not just the status-quo XEPs

nyco: because I think that a) the list is not complete, b) I'm sure others have opinions

I meant: should we add spam to our roadmap?

Ge0rG: the discussion is here on a more general roadmap, not just for the short term, but yes, having an idea of what should be in 2020 is an interesting topic in itself

I haven't followed the room lately on this regard, but stating things that need improvement is a grest step forward, would love to see this happen

great*

Ge0rG: but I'm not sure how 2020 could contain stuff that isn't mostly already there in some ways

ralphm: maybe CS-2020 should have a short-term agenda for 2021, and you prepare the long-term agenda?

I don't have any specific ideas yet.

It could. In any case, whatever both lists will end up being, I hope Council will also have opinions on this, before we say: this is it

Agree

ralphm, pep., FWIW, someone from iteam with powers on github is needed to grant permissions, I don’t think I have those permissions myself

Ge0rG: so my idea was sending a mail including those examples, and have a bit of discussion, getting to some common ground

jonas’: in that case, MattJ or Kev can help

then an email is better than a PR, ok for me

yay

speaking of iteam, https://mail.jabber.org/pipermail/standards/ is still broken

ralphm: yes, please do that

Ge0rG: noted. I'll raise it in their room

👍

5. AOB

?

Oh, I noticed that JC no longer has time to do the newsletter

Yes, wanted to talk with him

But basically I would like to discuss with commteam a bit this topic

And has announced that he's no longer doing that

Seve: sure, just a thing to mention

Anything else?

I'm part of it, and would like to see first what are we going to do

Nothing here

Splendid

6. Date of Next

+1W

I collect links... I wanted to write a bit today, I'll still be in support/help/contrib mode

7. Close

Thanks all!

nyco: also please mail the poll results to members@

Thank you guys, and sorry for my late show up

sir, yes sir

Better late than never

no pb

thx all

I'm on the minutes, then the poll results to members

general remark: I forgot to send out emails for the changes i pushed to xeps on tuesday, I shall do that tonight or at the weekend

jonas’: thanks!

✏

well done, Ge0rG!

both done

Nice

> I'm living between Munich and Stuttgart (South Germany) Not far from me

Hi everyone, I am implementing XEP-0313 Message Archive Management on the server side. Suppose the server supports offline storage (in addition to MAM) and the recipient of the sent message is offline. A message is sent to the offline recipient. Should the message be archived in the recipient archive WHEN: - the message is received by the recipient's server OR - the recipient turns back online

The former I think

Interaction between message archive and offline message store is something that I don't think is fully figured out yet

Thank you Zash , but that means that when the recipient client comes back online, it will probably pull the recipient archive containing the archived message while the server will be pushing the offline stored message. How will the client be able to know that the pushed message is the same as the archived message?

<stanza-id>

Ok I thought stanza-id were limited to XEP-0313. From my understanding stanza-id is set by the server. How can we provide the stanza-id outside of XEP-0313 IQ queries?

The way it goes in Prosody is: - Message comes in - Message is added to the archive - The archive ID is attached to the stanza - Delivery is attempted - If there were no online clients with non-negative priority to accept the stanza, add to offline store (with the stanza-id)

Not optimal, but it is the way it is because of historical reasons and the thing being modular.

So you are adding stanza-id to the offline storage mechanism even if it is not described in https://xmpp.org/extensions/xep-0160.html? Or am I missing a fundamental specification that introduced stanza-id for every message?

The module that handles MAM adds it when it adds the stanza to the archive. Then it just tags along through the rest of the chain of events.

Zash: when do Carbons happen?

Ge0rG: Somewhere in the middle

Before normal delivery I think

reda-alaoui: ideally, you could implement offline storage as a pointer into the MAM archive. When a client fetches from MAM, that pointer is moved forward. Under ideal conditions, you end up with the pointer after the end of the archive and no need to send anything after the MAM sync

reda-alaoui, XEP313 says you MUST add the stanza-id to every message that is archived

There are some corner cases with MAM limited to roster or clients only fetching a subset

So.. we're stuck on pretty basic stuff on slix while trying to add async functionalities in core parts. How do people even guarantee in-order delivery of stanzas? :x

so if you send a message that is in your offline store AND mam archive, but dont attach a stanza-id to the offline message

you are violating 313

pep.: you can't with MAM

because now you sent a message without stanza-id that is in fact in the MAM archive

lovetox yes I know but I didn't know it should be propagated to other specs

Ge0rG, not talking about MAM

Ge0rG, we’re more in an omemo case

lovetox: technically, you could have a separate store for offline that's filled before the message gets a MAM ID

But it's not really omemo related either tbh

Ge0rG : that's what we have currently

mathieui: stupid specs come with stupid consequences for the implementation

Ge0rG, it does not matter really, because in the moment you send a message from the offline store, you probably would put it into mam also

and then you have to inject a stanza-id anyway

reda-alaoui: but it helps a client tremendously to deduplicate, if the MAM ID is always sent

Ge0rG, its not about help

its mandatory

Ge0rG yes of course, it is a must have

ok, thanks a lot !

back to my code then :p

thing is a client cant differentiate between a offline message

and a live message

this means i receive suddenly messages from my mam:2 server that have no stanza-id, this would in best case drop a warning

actually just lets not test this :)

Implementation details galore

worst case is, i have to fallback to timestamp/body based deduplication

or even worse a client doesnt implement that because he only supports mam:2 and depends on dedup with stanza.id

Transition periods are messy

i have to implement that offline message purge

lovetox: the paragraph you quoted can be read as "the server must add the ID to the archived message only"

pep.: my outgoing stanza queue is synchronized

yeah yeah, i know we can read everything the other way, you are right its a grey area that was not thought about when spec mam

so yeah please server developer add stanza-id :D

So if I need stanzas in order I write them to that queue from the same thread and everything is fine

Daniel, how do you handle OMEMO encryption for exapmle, where you have to fetch devicelists and bundles and only then send your encrypted message

Would it be sane that if you set MAM defalut to roster, to then reject anything that doesn't go in the archive when you're offline?

As in, bounce with an error

pep.: call backs

> servers SHOULD include the element as a child of the forwarded message when using Message Carbons (XEP-0280) Another non-requirement.

Daniel, Ok so you block sending on the stream?

Well, yeah it's synchronized

Ge0rG: Make it a MUST for routing 2.0 or whatsitcalled

No I guess you could send other stanzas while waiting for the omemo fetches to return

Zash: will you implement that in prosody?

Daniel, and then you need to keep track of what came in what order, but you also need to prioritize the OMEMO IQs to be able to send that one message

Ge0rG: I make no promises about what I do with my free time.

I've just realized that smack will process messages asynchronously, so when I page through 2K of MAM messages, and update the progress counter accordingly, it will arrive at 100% while the async handler is still busy somewhere in the first half

update it on receiving the end of the page?

pep.: well I guess I can ensure order for a particular set of stanza but I don't have to. For example I can make sure that a leave and a join presence are always send after each other

Also the omemo message stanza waiting to be encrypted is not in the queue

Yet

In poezio atm, I am implementing the encrypt method as a stream filter, i.e., I want to catch everything going out to be sure none of what should be encrypted is sent in plain

I send plain text into the omemo pipeline and only after it was encrypted it becomes a stanza and gets fed into the queue

That means I can just send a normal message, and my filter will take care of the encryption

But that blocks the whole loop

> In poezio atm, I am implementing the encrypt method as a stream filter, i.e., I want to catch everything going out to be sure none of what should be encrypted is sent in plain Yeah I'm not a fan of that approach

That's what caused the gajim otr html body leak

hmm?

pep. on hitting send, i check if every pre condition is met for encrypting

The point is to prevent that

and if not, just nothing happens

text is still in input

Otherwise I have no guarantee poezio won't send plain stuff behind my back

i query whatever needs to be queried

(chatstates, etc.)

(and other plugins)

pep.: but if I understand you correctly what you are doing is creating a message stanza with a plain text body and then your filter removes the body and inserts omemo. I never do that. An omemo message for me is never a message stanza with a normal body ✏

Daniel because you dont have a plugin system

:)

Think of stanzas as being immutable.

Daniel, we're in python

Nothing is immutable

There are not really in my code but I treat them as such

You can still think of them as such

lovetox: yeah I don't like that kind of plugin systems

plugin system or not, I have to review every bits of poezio to ensure I won't leak plain. Stream filters were kind of the escape hatch that would have allowed me not to do that ✏

Daniel what you do you can only do if the client has knowledge about encryption

lovetox: yes. Obviously

the basic use case ("press enter") is fine, the issue is the rest of the commands and such (/correct, etc) which we have to check and disable

Also that means it doesn't make sense to implement OMEMO as a plugin in poezio anymore, without stream filters

yeah, Gajim has no knowledge about encryption, it just passes the stanza right before sending to the encryption plugin

And that's not possible license-wise atm

mathieui, correct is perfectly fine to use with omemo

lovetox, yes, but the point is we need to not forget any command which might send plaintext

yeah thats a really error prone way of doing this i think

instead poezio should not care at all about it

Which is why I went the stream filter way.. :P

do everything with the stanza as it would normally

and at the end omemo runs it through a whitelist of stanza nodes

lovetox, that was the plan

Is there an otr plugin for poezio?

yeah

Which is..

Because for that you also have to wait

Even for remote peers. Which is worse

I actually have no clue, I won't pronounce myself on it :x

but this has nothing to do with the problem pep. described i think

Daniel, the OTR plugin is implemented in the way I described above

you should have a handler that checks before creating a stanza, even before removing the message from the input field

if encryption is even possible

which is *really* ugly (whitelist elements, intercept the code that sends messages etc)

not pass it to haf of poezio than at the end find out we cant even encrypt because key is missing

lovetox, "before creating a stanza" doesn't map as you think it does, in slix

this has nothing to do with slix

that is UI code

you press the button

hmm?

on button pres, you check

> lovetox, "before creating a stanza" doesn't map as you think it does, in slix Then do it in poezio. Not in slix

nowhere I mentioned UI :P

I actually don't care about UI for now

This is the moment when you introduce numeric stanza filter priorities, so that you can do encryption last.

Daniel, yeah, slix/poezio ~

And then you need something *really last*

Zash: thus numeric.

pep., you check before the message would vanish from the input

priority=infinity+1

And then a plugin author thinks they must be laster than last and change the body post encryption.

Did I mention I don't like plug-ins?

Double-edged sword if anything

also writing a client is one thing

writing a client that perfectly interfaces with all kind of plugins

a whole other game :)

Just write a plugin management system and let the modules do the rest.

If you are okay with Lua, there is a decent plugin management system that's handling xml streams already.

Ge0rG: You know how Prosody is actually a Lua variant of node.js with some plugin management on top?

Zash: no.

Unheard of!

Zash: prosody requires a VirtualHost, so it can't be what you claim it is.

Daniel, yeah that's a different discussion, I don't have a choice anymore :)

In any case I suppose you currently have something of a `on_message` hook that has a message stanza as a parameter. What you probably want to do is create another hook `on_message_submitted` that has the plain text of the input and gets fired as soon as the user presses enter.

And if that hook returns true it gets into the normal pipeline and the input field gets cleared

And for omemo message you handle it differently

But then you need an internal representation for all message types, which is then mapped to the respective xml on transmission.

And then after omemo ran over it you can re-emit the message stanza and run it though the chain

Daniel, yeah, it's already what I'm doing but at the slix level. I'd need to look into it because atm it seems to me I'm going to run into the same problems

Same problem = leaking stuff you don't want to leak?

no

I lose in-order guarantees

Either that or I block my stream for ages while fetching devicelists and bundles

https://lab.louiz.org/poezio/slixmpp/merge_requests/24/diffs that's supposed to help here, but it's still not what we want