jdev - 2020-09-02

wait, hang on

a user just has one big MAM archive??

it isn't per conversation??

no

just one archive, but you can filter

oh yeah, filtering

right

was reviewing the Dino MAM code and it just pulls /everything/ from The Archive

which is an interesting way of doing things

eta, i'd assume most implementations do that

hmm, reasonable

Poezio doesn't, and tbh I'm not sure it will ever :/

pep., does it only do MAM when you hit page-up

just like mail clients sync their state with their imap server ✏

On join also

eta, does it handle gaps though?

And that's why, the first time you start those clients, it produces one bazillion notifications

Zash, yeah, that dino behaviour is very stupid

The smart thing would really just be to fetch when you open a tab, or to have something (that Inbox is.. somewhat?) that says "You have to fetch MAM here, here, here and here"

pep., also the annoying thing

Yeah well..

pep., I thought conversations did the fetch-when-you-open-the-tab thing

no, conversations does a full sync when it comes online

I painfully notice that every single time

oh

disappointing

Sync 😞

eta, why?

it is the right thing to do

otherwise you’re going to miss/delay messages

jonas’, I can't do things like open an old conversation and have it load scrollback

and users don’t like that

eta, huh?

it has the history locally

jonas’, I don't think it's the right thing to do as I said above :x

Or let's agree there are multiple rights

pep., so you prefer to require manual user interaction to get notifications for messages which wree sent while you were out of network coverage?

"to have something (that Inbox is.. somewhat?) that says "You have to fetch MAM here, here, here and here""

You can hint the user that there are unread messages in other tabs and that they can go look at them

jonas’, it doesn't, because I installed conversations after the time that this conversation happened

to do it per contact, is unnecessary complex, you have to reliaze that there are almost no mam messages if you start your client

eta, ah, AFAIK it’ll try to fetch stuff when you scroll beyond the local archive

on a normal connection you can download 1000 messages in about 3 seconds

lovetox, and fetching all the world is painfully slow and annoying for most people

there will never be 1000 messages waiting if you start your client

lovetox, tell that to my conversations.

and the gazillion of MUCs

of course MUCs are only fetched if you join the muc

^

the thing is, right

I don't actually care about fetching the world

But you do join MUCs at startup :)

bookmarks etc.

poof 80 MUCs opened, MAM fetching, uuuuuuughhhh

like you can totally do that in the background on a low-priority task or something

pep., I am not

that

I am in like five MUCs in C

the thing which annoys me is when it makes the unread counts go to like 10000 on everything

or generates notifications

really, what we probably need is a XEP for read state sync

eta, agreed

eta, agreed

Something that allows me to set the state in the past also plz

it doesn't even have to be that hard

an extra field in bookmarks + an extra field in roster

there will not be many messages in MUCs

you need a concept of a threshold

lovetox, tell that to my high-velocity IRC channels

eta, so you only want to sync state in MUC? Not 1:1?

up to you sync

that means a public muc has a threshold of 1

pep., and roster

so you only sync one day

roster? :/

a private muc has never many messages, so you sync everything

lovetox, that's your usage of private MUCs maybe

Not saying that I have, but I don't want to rule this out

thats why you can change the thresold even for private mucs, if you have that one muc where 1000 messages a day are posted

If Snikket decides that their version of private MUCs will only have few messages and that's how they do MAM, then fine. But I don't want XMPP to decide that for me

the one thing i learned is, you cant do this right without user interaction

its impossible

we need MUC categories!

there is no magic algorythm that lets you do this in all circumstances right

that way you can have a "low priority public MUCs" category

you can simply choose the threshold per muc

eta, you don't exactly need to formalize that. Especially since they'll be different for different design guidelines

true

hmm, what would be the ideal primitive for read state sync though

pubsub??

PEP probably

and how do you store that

on every received muc message update pep?

and if multiple devices receive it

all update it at the same time

sounds not really nice

lovetox: > there will never be 1000 messages waiting if you start your client You haven't experienced infinitely persistent MAM, have you? It's super fun when your archive goes back to the beginning of MAM itself.

does not mean you need to request it

But clients do this

And treats every message as new

but i talked about your user archive

not about mucs

mucs you need to manage per muc

So was I (personal archive)

in a user archive will not wait that many messages, that you have to manage it per contact

Well, start a new Dino profile and observe

Or maybe that got better lately? I don't remember

nope

thats why i sync only 7 days at first start

if the user wants all his history he can click a button that downloads it once

with a progress dialog

Yeah I'm not entirely fond of having arbitrary numbers like this. 7 days can still be empty and all of the chat happens on the 8th day or sth

then hit the button :)

Weird UX :/

not all, its easy to understand, and easy to implement

Zash for the moment I never cleared MAM history on movim.eu :D

its like whatsapp

you set up a new device, and it will ask you if you want to sync your messages from google drive or something

I got the impression that Conversations fetches stuff per contact when you scroll up a bit

lovetox, who says whatsapp is good :p

lovetox: the gajim history sync dialog is good, tbh

I haven't seen it in action, but from what I understand here, I'd rather fetch a set number of messages than a set number of days, at least

In poezio atm I try to have 2 pages of buffer at all times, so that I can display them instantly and only fetch previous to that

pep. here's an inbox prototype thing that's waiting for a client to prototype something against: https://modules.prosody.im/mod_map.html

k, I might have a look at some point

Slightly similar to Inbox right?

How do you count?

How how?

What's the count you're returning? What does it mean

Number of messages

what messages

All messages

All messages in the archive? All unfetched messages? All unfetched on this device?

All unread messages? (but fetched nonetheless)

It's essentially a MAM query that doesn't return the results, only a summary

It predates the whole notion of per-payload stuff that might be what people refer to as inbox

k

So it's all of the archive?

If I wanted to use this I'd mostly use @jid and <end/> I guess..

Maybe

Supposed to get you everything you need to show a "recent conversations" list

so on top of this we just need a "recent mentions / notifications" thing

Zash, only for 1:1?

Yes

It's a MAM query that returns a summary. Not a thing that fixes MUC

:P

Best I can think for unaltered MUC (with MAM) is to join with history(maxstanzas=1), which gets you some of the same data

reminds me to finish my protoXEP

not that finishing the XEP will change anything though >_<

i wonder how many people use Gajim

i just checked chocolatey, we dont maintain the package there, but it has 7500 downloads Oo

and thats a nerdy package manager for windows

Same person downloading it over and over again?

Or a single company downloading it for all their users? :P

hm no its 7500 since existence, the last version has 300

thats a bit more expected

lovetox: https://qa.debian.org/popcon.php?package=gajim might be of interest as well

damn

only rank 99700

need to improve that

“10:58:00 lovetox> there will never be 1000 messages waiting if you start your client”, tell that to my Dino, which I open maybe once a month, and is unresponsive for dozens of minutes while it fetches all of my pending messages.

SQLite being the bottleneck here.

Not XMPP or my ADSL connection.

really? sounds weird

did you ever google sqlite insert stats

or you have a really slow harddisk

https://stackoverflow.com/questions/1711631/improve-insert-per-second-performance-of-sqlite

I have a fast SSD, it’s limited by sqlite on a single thread.

there even the first try not optimized at all, insert 23000 records into a database

per second

lovetox, https://github.com/dino/dino/issues/853

Link Mauve: sqlite is not the bottleneck, it's just incorrectly used.

lets agree on, the bottleneck is probably how they use sqlite

:)

so far, 99% of problems with sqlite were at layer 8. It's one of the most robust pieces of software out there.

I didn’t find anything obvious last time I looked at Dino’s usage, maybe you’ll have more luck.

I’m… not very good at SQLy things.

Zash, lovetox, Pidgin just recently got installed more times than Empathy in Debian! https://qa.debian.org/popcon-graph.php?packages=gajim%2C+dino-im%2C+pidgin%2C+empathy&show_installed=on&want_legend=on&want_ticks=on&from_date=&to_date=&hlght_date=&date_fmt=%25Y-%25m&beenhere=1

never used empathy, seems it was pretty popular

It was preinstalled with GNOME.

But I’m sure most people just ignored it and installed Skype or whatever popular thing there was back then.

ah

Link Mauve: there are four indexes and four triggers on the `message` table. But maybe it's still accessing something index-less? dunno

Link Mauve: put your sqlite in WAL mode

will make Dino a lot faster