jdev - 2025-05-26

kinda a dumb question, but only one message / etc stanza comes over the wire at a time, yeah?

like it doesnt make much sense to thread / delegate the reception and initial handling of messages, because the code shouldnt be the bottleneck here?

Depends what you mean. It's a linear text stream yes so you can't thread the parser, but if you do a lot of work or IO for each incoming stanza it may make sense to background that depending

i guess in a way its kinda determining where to put the separation. im working in go lang which makes it pretty easy to just spin off tasks into their own thread, and database would likely get its own thread or whatnot

im working on making a wrapper library for simplicity, and as it is im letting the library handling the connection, decoding, routing, etc live in one goroutine and then basically sequentially handling each type of message in its own goroutine (not splitting by chat or anything)

you'll probably develop subtle bugs if a newer stanza processes before an older one, and they are related

jjj333_p (any pronouns), note that RFC 6120 mandates strict in-order processing of stanzas from a single stream.

so you may put different s2s or c2s links into different tasks/threads, but you mustn't parallelize processing of stanzas from a *single* stream

i figured it would only be a single stream

kinda what im doing now is the sdk unmarshals it, and does whatever internal procesing, and then it asynchronously handles off to a "user defined" handler

(you say unmarshal, I hope you don't use golang's built-in XML parser :-))

im making a wrapper library to an existing library, so its a couple layers removed

> (you say unmarshal, I hope you don't use golang's built-in XML parser :-)) nah its using what mellium provides ↺

ah well, then you can shout at mellium if you run into issues, that's fine :D

mellium does pretty good at having all the different pieces, but it doesnt really have anything to glue it into one user friendly piece

so "asynchronously handing off to a 'user defined' handler" is likely problematic

I'd expect a go channel per stream where you get stanzas.

that would be the correct primitive.

if i do that the entire application becomes single threaded, i have to decide some point to break

what application is this where that matters?

im planning to incorporate this into a client, so you would expect the ui and db to be kinda asyncrhonous from handling messages

im exaggerating more than anything but still, feels unnecessary

well sure you can run UI and DB in separate tasks, and you might even gain something from that

like, the UI may have to query the database unrelated to XMPP events.

but the XMPP events should be processed in order (though that matters less for a client than it does for a server)

I suggest to have an XMPP I/O task, a database task, and a UI task, and couple them via channels of some kind.

where it does matter a lot in a client is, however, message processing.

my message is too long

you'll end up in all kinds of fun hells if you don't process <message/> stanzas in the exact order they were received. ✏

> you'll end up in all kinds of fun hells if you don't process <message/> stanzas in the exact order they were received. oh im talking past decoding, this is strictly for "userland tasks" ↺

i tried to send context, but i have to go pastebin it

jjj333_p (any pronouns), I'm also talking about userland tasks

think OMEMO

or MUC for that matter

or anything related to messages.

you don't want to accidentally reorder messages in your view.

users really hate that :)

ah yeah i wasnt considering those to be "userland"

what _is_ userland then? :)

i kinda considered decryption to be apart of the parsing

> what _is_ userland then? :) i have to fallback to my js roots for convenience but think ```js client.on("message", event => { event.reply(`recieved ${event.body}`) }) ``` ↺

here, for context. all the way up until that go routine call at the very end everything is within the same goroutine as the connection https://pst.everypizza.im/2SPQ

> i have to fallback to my js roots for convenience but think > ```js > client.on("message", event => { > event.reply(`recieved ${event.body}`) > }) > ``` to clarify, i consider "userland" to be where the user does actions with the end data. i would kinda expect the library to handle decryption and whatever else and be mostly opaque to the end user ↺

i.e. you kinda have a "client" object in your code that you can then probe actions on and get messages out of

> I'd expect a go channel per stream where you get stanzas. do you mean a goroutine? ↺

because i started out using the mux to hand things into a channel which then each message type got handled syncronously, but i got yelled at in the go room because either youre blocking the "main" routine or youre using a buffered channel which is aparently a nono

> because i started out using the mux to hand things into a channel which then each message type got handled syncronously, but i got yelled at in the go room because either youre blocking the "main" routine or youre using a buffered channel which is aparently a nono It's totally fine to handle stanzas synchronously in a goroutine, that's basically what it was made for 😁 just don't do it in the main goroutine ↺

Hello

> It's totally fine to handle stanzas synchronously in a goroutine, that's basically what it was made for 😁 just don't do it in the main goroutine i mean the whole library / connection is going to end up in a goroutine, because the ui lib i wish to use demands you invoke it from the main goroutine, and then all operations basically have to lock to that goroutine ↺

Ah, that second lock is unwelcome 😄 ✏

can't you write all changes to db and then lock on the db, as usual ?

eh it makes it fairly easy, it comes with a built in .Do() which will just silently add the operation to the queue and move on, or .DoAndWait() if you absolutely need to wait for some reason

> can't you write all changes to db and then lock on the db, as usual ? i was going to keep the db kinda in its own synchronous goroutine, keep everything in its own worlds ↺

gui gets a goroutine, db gets a goroutine, network gets a goroutine

i did learn that unless you give a channel buffering, it does block until the item is recieved, so you can use insertions to a goroutine to queue read operations, or something of that sort

i wish to avoid blocking operations as much as humanly (or machinely?) possible, because i have gotten tired of apps that hang when they need to read something from db or fetch a network resource, or wont let operations go through because the ui needs to update (i have seen all of these)

Just spin a separate thread for UI

Too bad, the nice thing with goroutines is instead of splitting on domain (as you do) you can split on functionality (get message, send message, post to node, ...), block inside functionality but have functionalities run separately

this is possible because goroutines aren't threads/processes, they're extremely lightweight concurrent contexts of executions

Do you really need to parallelize, for example, DB writing and stanza processing from the same entity? In most cases, I don't think you need that. You usually want one to finish before the other, both to limit the number of states your program can be in, so it is easier to reason about, and also to provide back pressure.

> Just spin a separate thread for UI go doesnt work with explicit threads, and the ui library will panic if its not the main goroutine (can be interpreted as the world thread) ↺

so i just delegate _everything else_ to goroutines

> Do you really need to parallelize, for example, DB writing and stanza processing from the same entity? In most cases, I don't think you need that. You usually want one to finish before the other, both to limit the number of states your program can be in, so it is easier to reason about, and also to provide back pressure. you can use mutexes and other threadsafe operations to kinda lock and await for other threads to get done with things ↺

so that way you arent prevented from downloading something while the db is writing something irrelevant

>> Just spin a separate thread for UI > go doesnt work with explicit threads, and the ui library will panic if its not the main goroutine (can be interpreted as the world thread) I think all major ui libs have a method You can call from any thread that will schedule code to run on the UI's thread. idle_add or smth like that. ↺

> I think all major ui libs have a method You can call from any thread that will schedule code to run on the UI's thread. idle_add or smth like that. yes i explained that earleir this one has that ↺

but you have to execute window.ShowAndRun() from the main routine which blocks until it closes

> Do you really need to parallelize, for example, DB writing and stanza processing from the same entity? In most cases, I don't think you need that. You usually want one to finish before the other, both to limit the number of states your program can be in, so it is easier to reason about, and also to provide back pressure. It's not about parallelizing, it's about contextualizing "things that make sense together". Not a write from a db and an update in the ui, but a click on the "send" button and a click on another chat, for example. ↺

I gotta say that doesn't make a lot of sense from a client point of view, think about this common thing that happens: > user writes a message in a box, hits send

You can't fire that off to independent workers, one who writes to the database and one who sends across the network, because the network might be down or the process might be killed after network send and before database write succeeds

Sounds all horrible, it's complicated enough to find out what's going on in a fairly advanced client which can reach 100k loc easy

If you have x asynchron routines/threads I think you will be the only person to touch that project. Just my 2 cents

Frankly this is why general libraries never get used by clients, too many assumptions that prove incorrect in real world use

In Gajim everything is in one single thread, UI has high prio, network lower prio, if there are CPU intensive ui operations, pulling from the network simply gets Stalled a bit

Except of course stuff like reading big files or downloading files, these are put into small shortlived Threads for the specific task, but they are not related to the xmpp stream, think download file with http, load Avatars From Disk at start

Most ui blocks are because the UI has too much to do, and the UI can not draw in another thread as you discovered, needs to be in the main thread

> You can't fire that off to independent workers, one who writes to the database and one who sends across the network, because the network might be down or the process might be killed after network send and before database write succeeds mine puts it into local echo (which is a synchronous in-memory object) and network immediately, and once I hear back from the network it pulls back from the local echo, and writes to the database. The database write is completely async and unawaited (though with an error handler attached for debugging, in case something exotic happens), so the user can go right to composing the next message ↺

halscode: so what happens if they write 3 messages while the network is down and then shut down the computer ?

> halscode: so what happens if they write 3 messages while the network is down and then shut down the computer ? right now, enqueued messages are lost. it's pretty predictable from the end-user side, since the local-echoes that haven't been acked yet have a pulsing effect. Persisting it to disk, especially when I expect it to not stay in local echo for very long at all during normal use, is a real waste of resources. Though thanks for pointing that out, that is still an area I can improve on ↺

edge cases do need to be considered

It sounds like sometimes they've sent over the network and sometimes not and they might end up in the database and sometimes not

oh also, I think if the network is down it'll immediately fail, that's another thing to fix... in that case, the queue would be immediately persisted to disk and removed from the database once you're back online and it's sent. Also requires more explicit messaging than just "waiting to see if it's been sent or not"

Basically you need to decide if you want at-least-once (meaning sometimes more) or at-most-once (meaning sometimes never) delivery, I think without doing crazy mam things you can't get exactly-once delivery

If it fails to deliver, it fails with an error message, or it's the/a server's fault for fumbling the message and not mine lol

and some retry conditions, though I'm not sure I actually have that

I suspect for a client you'd want at-least-once, which implies commit to database first with a flag that means not-sent-yet, then keep trying to send and once successful flip the flag to sent

that's true, but I also don't want to write to disk and then delete it again in .1s if it fails ✏

If it succeeds? But also why not?

disk i/o abuse, mostly. though, ugh, errors of type cancel shouldn't be that common so it's not a very important case

disk i/o isn't really going to be a consideration here, I would be much more annoyed by lost messages

I think part of it was that originally I wasn't using a proper database so I couldn't just delete failed messages, I'd have to reconstruct the whole conversation history to just leave that one out

You almost surely don't want to delete them, you want to keep retrying or mark them failed so the user can retry or at least know they didn't send no?

They shouldn't stay permanently, they're failed, they're irrelevant to the rest of the conversation. Currently they are shown in red next to an error message, until you restart the app. (I think app/window close would be a good time to clean up failed messages from the database, so it doesn't disappear while someone's reading it.)

so I'm trying to send an important message and having trouble getting internet to work and reboot my device in an attempt to fix it and the message(s) are just gone? I would hate that

This actually has happened to me but with Conversations they are marked not sent with the error and I can select "resend" if I want ✏

> so I'm trying to send an important message and having trouble getting internet to work and reboot my device in an attempt to fix it and the message(s) are just gone? I would hate that yeah, there's a reason I haven't published this thing yet 🥲️ ↺

These are good conversations to have, for whatever reason it's something I hadn't considered, so it's good that it's being brought up. Thanks, moparisthebest

Sometimes you gotta have someone play doubles abdicate with you

So maybe it only gets auto-deleted on explicitly cancel-type errors, that can't be retried (network errors would be treated like "wait"). You can't resend it if, say, you tried to send something to a MUC you're not allowed to send messages in, or if your contact doesn't exist for some reason ✏

To complicate things more I'd say not even then, I often send messages to a MUC and it can't be delivered because remote-server-not-found , because they are having maintenance or some other thing, and again it's nice to long-press and click resend when they come back

> These are good conversations to have, for whatever reason it's something I hadn't considered, so it's good that it's being brought up. Thanks, moparisthebest 👍 ↺

Yea it's hard/impossible to think of all these things unless you've experienced them lol, happy to play rubber duck who's seen things 🦆☠️