jdev - 2023-07-20

Not that I don't trust edhelas, but better safe than sorry, so… Right now "bridged contacts" have this identity (for some, "pc" is replaced by "phone"): ```xml <identity category="client" type="pc" /> ``` Is it still correct and am I going to break other clients than movim if I add another identity of category="gateway" to them, eg: ```xml <identity category="client" type="phone" name="Slidge (Telegram)" /> <identity category="gateway" type="telegram" name="Slidge (Telegram)" /> ``` ✏

You don't trust me :'(

but its not a gateway ..

gateway should be on the component that *is* the gateway

Ah !

just my opinion, but i see what you want to do

you want to discover that a contact is connected via a gateway

But as a client I'd like to know if the presence of the contact is actually from a gateway then 🤔

Then i can display a nice icon or something in the UI

yeah i guess as long as both identitys are there its good

just make sure that the actual gateway is discoverable *as* gateway

and not is the same as a contact

i also dont see any other category that could be used

> When multiple identity elements are provided, the name attributes for each identity element SHOULD have the same value. ✏

seems important to note from the standard

not sure why though

oh, good catch, I was wondering about the name, but I happily respect this SHOULD

the component has gateway+account+pubsub+conference identity

thanks

is this for disco to full jid or bare jid of users?

full jid

i would assume both, why would there be a difference?

A big difference

not sure what I return on contact bare JID

let me check

because type=client is full jid, bare jid should be type=account typically

ah, ok yeah clients never deal with querys to bare i guess

so this would be needed to be answered by the gateway or server

well I reply the same on the bare JID too, so I need to fix this I guess

just identity type=account and no features is what it should reply?

because features are per-client, right?

except you have features on the account, like pubsub

(what we typically call PEP)

does anyone know why the name attribute must be the same?

redundant duplicated redundancy everywhere?

probably because there should only be one name

also I would assume an additional category=gateway to be fine on the bare jid?

I wonder why you have to include the `jid` attribute on pubsub node listings, where *everything* has the same jid anyway

I also wonder why this doesn't apply to the `node` attribute

Just XEP-0030 things 🤷️ ✏

interestingly https://xmpp.org/extensions/xep-0163.html#example-5 has "client" on the bare JID (examples not normative, got it)

that's a full jid (from=romeo@montague.lit/orchard)

oh yeah, confused from and to, sry

larma, there is not type=account, you meant category=account=?

yes, category=account and category=client was what I meant in that message

why did anybody ever think that using category+type was better than type+subtype or category+subcategory? 😀 ✏

larma, does dino allow backwards mam querys in MUCs? like on scrolling upwards?

so for bare JIDs (of *contacts*), does this sound reasonable? I implement a minimal subset of PEP so that the contacts can have nicknames, avatars and vcards ```xml <identity category="account" type="registered" /> <identity category="pubsub" type="pep" /> <feature var="http://jabber.org/protocol/pubsub" /> <feature var="http://jabber.org/protocol/pubsub#retrieve-items" /> <feature var="http://jabber.org/protocol/pubsub#subscribe" /> ```

oh, and the same gateway/telegram identity too

> larma, does dino allow backwards mam querys in MUCs? like on scrolling upwards? I'm not larma but I don't think so, from my tests on muc join dino just requests 20 messages and that's it? I implement the <flip/> thing but I don't think I've found a client to do "real life tests" against

nicoco, the flip page thing is quite important, when i implement backwards queries in gajim, i will only do it for services that offer that feature

cool, hopefully I'll discover that automated testing covered it successfully. and if not, I'll fix it :)

lovetox, why is flipping important for backwards sync?

it makes the implementation much easier, because you can request history without dealing with holes

apart from that its more efficient

I don't see how it makes a huge difference. You request a page of say 10 messages. With flip-page, those messages on the page come in a different order, but beside that, nothing changes. Reordering the messages on the client shouldn't be too hard either (just wait for all messages of the page to arrive, flip it, continue processing as if you got the page flipped initially)

yeah thats, it you had to implement a whole mechanism where your request multiple stanzas, wait for completion, reinstert them into your pipeline so they act like "just received" messages

but yeah if you did all that, and dealt with all issues of that design, i guess it does not make much difference anymore, apart from beeing simply slower to display messages

I mean, I understand it's useful, but I wonder why it makes more of a difference for MAM upwards scrolling than for syncing history with the user's own MAM server

And if you already have the code and logic to handle servers that don't do #extended, it's really the same code and logic for MUC MAM servers as it is for the user's own server

i would never implement that for the users account server

downloading tousands of messages from your homeserver takes seconds, instead of having that complexity i simply ask the user if he wants to have a full sync with his home server

apart from it beeing useless as everyone using omemo anyway

But I mean, you had to write that code for handling holes in your local database anyway for syncing with your home server, no?

So why not use the same code when fetching from MUC MAM servers

what holes?

you request messages from your server from the last known stanza-id

how can there ever be a "hole"

uhm, ok, I mean you *can* do that

uhm, you must do that, otherwise how do you tell the user he received a message :D

but that means you'll have to wait several minutes if there is a lot of history since you logged in last time just to get the latest messages, which are probably the ones that matter to the user *now*

several minutes? to download a few MB, did you test this on a year 2000 modem connection? ✏

We changed our code for that because it *was* to slow

because what matters is not how many MB it is, but the latency of the server responding to your requests

yes sync needs longer on a slow connection, as does browsing the web, i think users understand that

because each page needs at least one full roundtrip

luckily we can choose the page size

not arbitrarily

You can choose the page size you request, not necessarily what you'll be given.

i wonder larma, did you feel it was slow, or did users complain?

both

and now you fetch an arbitrary amount first from the server, and later load more?

that was my question yesterday, how do you keep the messages in order in the local db?

you insert later messages first into your db

and afterwards earlier

on what exactly do you sort before you display the messages to the user?

your MAM server puts a timestamp with the time it received the message. If you sort by that it's the same ordering as if you ordered by arrival time (ignoring server time not being synced properly) ✏

and you did test this, when multiple messages have the same timestamp?

because i found no solution for that problem, with mam backfills, messages with the same timestamp will be in the wrong order

it works fine for a forward query, because you can sort on insertion order IF timestamps are the same

but with a backfill, you would need it the other way around

but even with backfill, the messages are in order and the pages are in order, so you do still have the same strict ordering on the messages

yes if you receive them they are ordered, but the question is how can you preserve the order when writing into the db

if you do backfill query where the first message has timestamp = 10, and you insert it into your db, and afterwards make a backfill query where the last timestamp = 10, and insert that again

you will have it in the wrong order inside the db

or do you do backfill queries with timestamps?

and ensure so that if you receive something you receive all messages from a particular timestamp

nicoco: I would have to test, but I bet adding identity of gateway on something that is not a gateway will confuse the heck out of my app

It will probably list all your contacts as gateways then in the "add new contact" dialog

We already have client/sms so if you need this info at that level without doing disco on the domain could register client/telegram etc why not?

singpolyma: Huh! This is the sort of thing that I feared. edhelas, thoughts?

the question is not if it breaks *some* client

the question is, can a gateway identity be attached to a client or account

is it technically legal? if so, better to find and fix the clients that break, or do nothing and let them continue to be broken?

i would say if we say, no, this would limit us to express what that client is, and would lead to worse UX

I think the registry is pretty clear

so i tend to want to say yes

Gateway means it is a gateway

And as I say we have evidence of this with client/sms already existing in registry

singpolyma, that would be wrong

or lets say we would lose other information

for example the nice information that a client is a phone

What would be wrong?

You can have client/phone and hypothetical client/telegram both if you know the client is a phone

you mean attaching multiple times the same category?

Yes

i dont think this is intended

all types are either/or stuff

Except when they're not, as here

FWIW I'm OK with whatever the tribe decides. I agree that the proper question is whether this is legal/correct or not.

nicoco: it's legal, just doesn't mean what you want it to mean imo

I'm not sold on adding all gateways into the client category

One could argue that we don't need it at all and clients should simply disco also the domain to find out if it's a gateway

This should be easy and need to be done only once

Certainly that would be my default opinion as well

I only mentioned that client/sms exists as prior art indicating the registry may lean the other way. But disco on domain is a very sensible option

Usually with a gateway the domain will be in roster anyway, in which case you can even use caps

I don't have a strong opinion about what the correct behaviour is. Also, I don't develop any client so I'm inclined to listen to what client developers prefer. FWIW, before edhelas asked this, I had understood the same way as singpolyma, ie, identity=gateway means "I'm a gateway service" and not "I'm a _puppet_ (=bridged contact)".

singpolyma, i thought about what we talked about with the negative incrementing field, when backfilling, i think this works good for the MUC case, where you want to get something before your first join

but with MUC, we have the situation, that a user might not join a very long time, and when we join we dont want to get all history, maybe stretching months

so naturally a hole is created

to backfill a hole you cannot use the negative incremented field ..

u64, anyone?

lovetox: a hole might need a tuple, yeah

I think you want to record 'holes' outside of the storage of messages. Personally, I'd do it by the IDs of the surrounding messages.

And for sorting of the message list, I'd do something like a u64 based on the time of the last message in a page in the high bits, and doing up/down from there in the low bits, or similar.

(Which allows sensible fiddling then, if the time sequence breaks down)

Kev: what's a u64 can you point me to something, my Google foo is not good enough

singpolyma: a tuple of what

lovetox: well, the problem is you have messages in local archive let's say you've given sequence number 100. Now you get new messages but you want to page backwards and maybe not get them all because the hole is so big so you don't know how much "bigger" than 100 to be. So you assign 101,0 then 101,-1 then 101,-2 all the way back when paging backwards these always sort greater than 100 due to 101 but are internally ordered by the second number. Forward paging from these resumes at 102 etc

(100 in this system is implicitly 100,0)

> : what's a u64 can you point me to something, my Google foo is not good enough 64bit unsigned int.

Kev: right. You're suggesting bit-packing the tuple, effectively

I don't think I am. I think I'm suggesting that the holes be recorded in a different table (assuming SQL) from the messages themselves.

The tuples can simply be hole=(last id before hole, first id after hole).

It's not about recording the holes, it's about how to make the ordering work out without renumbering

The u64 thing was just a way of ensuring that when you fill holes you still have room in your sorting sequence for the messages to be in the 'right place'.

Right

Ok thanks for the input, will try to work with this

Hello how are you doing everyone