XSF Discussion - 2018-11-06

lovetox: I'm not sure if there are situations where Gajim is using an ID generated by another entity, like e.g. Last message correction (where it's not a security problem, but to illustrate the attack channel)

What's the official way to talk to iteam?

I'd like to get https://www.mediawiki.org/wiki/Extension:MobileFrontend installed/enabled on our wiki

join iteam muc

I still don't receive emails from the members list. Can somebody double check, please?

I'm not sure if my new email address was added.

Seve: which email address? i can check

Alex, seve@delape.net, I used to have my JID the same as my email address before in case you see that still there

Seve: its there, but sais `nomail, reason B`

looks like you have disabled it once. I can revert he checkbox

you also maybe alble to congigure it here yourself here: https://mail.jabber.org/mailman/admin/members/ or send commands to the list. Just let me know what you prefer

That's weird... Yes please do, Alex. Very appreciated!

resetted, I think you should be good now

Alex: thank you!

Alex, I have no password to edit it myself. I tried to see if I could ask for a new password and it says `Your subscription request has been received, and will soon be acted upon.` I wonder if that has some relation to it?

Seve: lets see if you get new messages to the list. If not then please contact me again and we try to figure it out. To me it looks all good now.

Sure, Alex. Thank you very much for your time, I will let you know :)

in-band bytestreams. is there a way to close an IBB with an error while the peer is *not* sending to you? ✏

jonas’: remember the last incoming message id?

jonas’, can't you always send a session-terminate (if we are talking Jingle based IBB)?

flow, no, plain IBB

jonas’, AFAIK you can’t even do that if they’re sending to you, since they could use the <message/> transport.

<message type='error' id='...'>

Speaking of which, can we get error messages stored in offline storage, because Carbons and multi client?

Yes

MattJ: In prosody 0.10?

"LOL"

Ge0rG: You mean in MAM?

Holger: MAM is not needed if you have per device offline storage queues

Hah.

Why proper solutions if we can have ugly hacks!

And my evil secret plan is to defer implementing MAM in yaxim (and annoying MattJ with message loss) until he gives up and finishes client tracking

Holger: MAM is an ugly hack.

How do you handle outgoing messages with offline storage? Wait whether the client enables carbons and then push them out?

How do you do paging?

different use-cases

Hm?

although

ignore me, I want to see how this plays out

Holger: how do you handle sync of messages between MAM and offline? Just drop all offline messages before sending presence?

That's what Conversations does, the alternative is de-dup, obviously.

Whatever that has to do with my question.

I think MAM becomes problematic to implement if you *don't* want full-sync. If you do, like you'd get with per-device offline storage, it's trivial. I don't get why you wouldn't just want to implement this.

Holger: MAM is *also* a dirty hack. Is it better now?

Ok, so that's go for a worse hack instead.

Holger: because it's complicated with my current code base

Holger: MAM is the worse hack.

Why?

Actually sending out Carbons from offline is a I nice idea

Haha.

Holger: it's full of race conditions!

For example?

You don't need paging if you do full sync.

Holger: between querying MAM and dropping offline history

It's dirty hacks all the way down to the silicon

Holger: also between completing your MAM sync and receiving first live messages

Ge0rG, isn’t bind2 there to solve all those races?

jonas’: yes

purge offline, enable carbons, give you the id of the last stanza in MAM

jonas’: also magically enable / resume SM

Until we have that, de-dup should be trivial using stanza-IDs.

Holger: you mean unique ids?

Will MAM inject ids into offline storage?

mam:2 guarantees that, yes.

Fwiw Conversations would handle the dedup just fine. But I don't like the extra traffic

But if you just drop offline history, you'll get message loss if your MAM is "contacts only"

Yes, I don't know why we offer that option.

Yep

And dropping before MAM sync is lossy

Or disabled

You get even more message loss if MAM is disabled.

Right.

I'd like the mam preference to be announced in disco

That could help circumvent that problem

daniel: you drop offline history if MAM is disabled???

I'd like to see the "contacts only" option ditched.

Ge0rG: yes

daniel: ...

> You don't need paging if you do full sync. You might need it for traffic throtting. Otherwise the server will flood you with messages and if you're not fast enough with acknowledging it, you'll exceed 0198 queue size limits and the server will kill your connection.

*throttling

I open yaxim once in a few weeks, it would become unusable if I implemented that thing.

Holger, FWIW, Conversations *is* unusable when you open it only once in a few weeks.

and Gajim

or even once a week

I plan for like one hour or maybe two for it to sync when I haven’t used it for a week

jonas': Hm no I didn't run into that.

maybe it’s just me or my device?

ever since that full text search index each message takes like 0.5s to process

you can do the math with a few public MAM’d MUCs.

Zash, Gajim fixed that (for public MUCs only) in their latest release from today.

it’s terrible in any case.

Link Mauve: By some random coincidence, I can't start Gajim anymore

XMPP is terrible.

fg

That may be related, you should report this bug at gajim@conference.gajim.org.

Maybe they’ll do a 1.1.1 fixing it.

I think it's some Python search path thing

yes the user can set a threshold now for public mucs how much gets synced

and the default is 1 day, if you join 20 mucs, you still can expect around 30 seconds until everything is joined and everything is synced

problem is, you absolutly have to sync everything at start with MUC

otherwise you just plain accept not delivering messages to your user that he might want to read

im not seeing how we can optimize that in any way

The problem is that 1 day is pretty random. On some days you just have one message in the last 24h on some days you have 1000

I'm considering - for public mucs - to just ditch the entire history instead

Then normal backlog scrolling works as intended

Instead of trying to fill gaps or what ever

But nothing is ideal. There is no good solution

so not storing messages in public muc to db, and making it completly load on demand

Yes

yes the only way back scrolling can work i think

*in public with mam

but this is probably acceptable for a mobile client, but if i do this in Gajim people will scream

Why?

they have there 200 MB "i have logs back to 1990" dbs

and want to search them

but yeah indeed nothing speaks against making it default not storing, and if a user wishes he can store for some mucs

but then no backscrolling there

maybe thats a good solution for both worlds

Yeah I haven't thought this all the way through yet. It's just that idea I'm having

Especially for mobile and like short outages you might still want to have a local history

But certainly not catch up with a week or so

and the whole backscrolling thing can get pretty easy if client devs could depend on the server using sequential mam ids

then its trivial to fill a whole

i still think whatever reason that was why this was dismissed in the XEP, was not worth it at all

but mam exists a long time now, maybe we should ask around, and if there is no implementation that depends on mam ids not beeing sequential (because they technically can not do it) then just ditch this, there is no need for something that limiting us, that nobody plans to use anyway

It's complicated on a database level

Either pretty expensive or impossible in clusters

hm doubt it seriously :D

its just that it would be probably considerable more work to make it so

but then again, where are these clusters, are we writing xeps for that one provider in X years that sets up a cluster

or for the 1000s who do not

but i guess we had this discussion, the conclusion was, add a disco feature

"sequential-ids"

if i see this i offer back scrolling, if not i dont

Well there certainly are clustered setups in practice and I wouldn't want to tell people "sorry XMPP doesn't support this".

is ejabberd not a bad example, as you offer sequential ids :D

lovetox: I forgot, you actually need sequential IDs or are increasing IDs (i.e. ejabberd's timestamps) good enough to solve the problem?

If the use case is to know the gaps you need sequential and not just increasing

maybe im getting my definitions wrong, but sequential and increasing are not counterpars

lovetox: Yeah these days we just ignore the clustering issue because it's somewhat academic in practice. With microsecond timestamps you won't end up with non-unique IDs, or if you ever manage to, then wow you have a message loss/duplicate. This is XMPP after all.

And that's complicated even on an single db

I mean can you tell me the sql command for insert with id=max(I'd) +1?

Don't SQL engines have this built-in? But that'll obviously be expensive on a cluster, yes.

I think the bad ones do

lovetox: Sequential is 1,2,3,4, incrementing is 1,4,394,1039.

then incrementing is enough

But from my understanding you try to avoid this in db design

lovetox: how do you find gaps then?

yeah this fixes just the message ordering problem

you still have to track gaps somehow

I started writing a thing based on linked lists once

So what problem can I solve by tagging each message with the id of the previous message?

You solve the same problems and run into the same problems as with sequential IDs, no?

Given a bag of unordered messages, you can shuffle them into sequences, no?

I guess what you do these days is DAGs, i.e. you can add a merge message that has multiple parents once the cluster notices the conflict. Leave it to clients to cope with it.

the problems we are facing are two fold, 1. have a way to determine where you have a gap in your local storage and fill it efficiently by requesting from mam 2. If you start your client, display all messages in the correct order

2. can be solved by just increasing IDs, or just microtimestamp

> 1 day is pretty random

make mam allow query the last N stanzas?

flow: ... <before/N>

we can do this already

wait no, before="" max=N

high resolution timestamps has problems too

like like being unavailable from C stdlib

maybe the server can solve 1. for us, what if we have a query i give 2 mam-ids, can the server tell me if there are messages in between?

and also having no guarantee of being strictly increasing

lovetox, then why 1 day and not, say, 100 messages?

Zash its enough if its most unlikely to produce duplicates

i can live with a duplicated timestamp 1 out of a million messages

Holger, are you talking about Matrix? :)

but serious, this sounds appealing

Zash: The C standard doesn't return the current time at all, no? So you can't implement MAM with it anyway.

flow it can be done, but its not the same thing, last 100 can only be done by backpaging

you get the last 10

then 10 before that etc

flow: Yeah Matrix and I think various other distributed DBs are doing it that way.

while from a date, gives you all messages orderd from old to new

And all those DCVSes ...

and this is pretty important, because i need the messages sent ordered to me

lovetox, ahh so MAM is missing a "give me the last N messages starting with the oldest" query?

i dont know if it needs it .. but yes i guess would be nice

(I think we possibly have been there before)

i more and more think, maybe we dont need this at all, internet gets faster and faster, if i join a muc i show a loading screen and count the messages with a nice UI

flow: I don't think MAM is missing query related things like that

so the client needs 30 seconds, if you dont start it a week, so what?

show me an application that does it much better

Zash, possibly, possibly not

lovetox, na, that's not what drives me, but I can see why one could be happy with that

Fetch the last chunk of messages and show them. Then do the sync thing quietly in the background.

Zash, fetch using <before/> and max=N?

flow: right

i could pull with before say 20 messages, but dont store them to db then sync the db in the background

Dunno, I think you run in all kinds of oddities with current MAM

but there is no "background"

there is only one connection to the server

and it will be slow to the user if i fetch stuff contstantly

it would be nice to know what the user wants to do and priorities that

but he could switch to any muc instantly and then i need the full stuff there

lovetox, do small chunks of background sync and pause them once such "high priority" things happen?

and all that seems rather complex, instead of just a loading screen

:D

or just do small chunks and see if it turns out to be a real issue

before we are talking about potential solutions for non-issues ;)

joining 20 mucs is also slow if you dont fetch MAM at all

It is

hmm, dunno, my poezio joins the ~15 MUCs reasonably fast I'd say, but YMMV

yeah can you give me a time on that?

just so i have something to compare Gajim with

lovetox, a few seconds I'd say, but it is really hard to tell as the UI elements are already seen even if the MUC is still in the process of joining

hm other topic, nickname conflicts when joining a muc how do clients handle these

gajim does a popup and the user has to choose another nick

but im thinking i dont want to ask the user anymore and just join with a added "_" or something like that