jdev - 2020-04-17

hm lib dev question

if a lib has a JID object, and has a getBare() method

what should it return if the jid is only a domain for example "asd.com"

would be wrong to return asd.com

should it raise an exception, NoBareJID or something?

lovetox, why would it be wrong? asd.com is a valid bare JID.

or is the localpart not mandatory on a bare jid

ah ok nice

asd.com/foo is a valid full JID.

so barejid, is just without resource

Yes.

ah k thanks

that ^

lovetox, http://jxmpp.org/releases/0.7.0-alpha5/javadoc/org/jxmpp/jid/Jid.html https://github.com/igniterealtime/jxmpp#jxmpp-jid

how likely is it that 2 different hash mechanisms produce the same hash

like entity caps its mostly sha-1

but theoretically you can also use something else

right now i store the hash mechanism beside the hash

lovetox, XEP-0390 fixed this issue AFAIK.

but i wonder if i can just not store the hash mech

lovetox, it is unlikely. but it’s stupid to not store the mechanism, too.

it only costs a few bytes

Can get down to a single byte if you convert the string into an enum.

Assuming you know the mechanisms you can use.

there was a thing...

its not about storage cost

i have a 115 cache

which constists of hashmethod/hash -> discoinfo

but then there are entitys we have to query and cache which provide no hash or have no presence at all

like for example a muc

i need to also store these disco info data

but its hard to use the same cache

because for muc i need a JID -> DiscoInfo kind of cache

and that i need 2 different caches for discoinfo data .. somehow i dont like it

but that's how it is

lovetox, in aioxmpp, we listen on presence info and prime the JID -> DiscoInfo cache from the hash -> discoinfo cache

or, are you thinking about a generic String→ disco#info cache?

that means that discoinfo lookups themselves only ever use the JID -> DiscoInfo cache ✏

guess one could do that, but I'd personally wouldn't do so

which is pre-populated on the fly by the entitycaps component which listens for presence

especially since you could persist the caps cache to disk, while the jid→disco#info cache is somewhat unreliable and should have a sensible timeout (smack uses 24h)

ah nice so when you need caps, you always access only the JID -> Disco info cache

thats exactly what i was searching for

lovetox, correct

thanks

jonas’, does that jid→disco#info cache only include values obtained via caps?

flow, no

or do you also put in results of actual disco#info requests?

in fact, the jid->disco#info cache is in reality a JID -> Future(disco#info) cache

yes thats the goal flow, often i have to disco info instances who dont have presence

and i save that in a cache and store to disk

lovetox, I’m not sure storing the jid->disco#info cache on disk is a good idea

store to disk ephemeral data?

for example a transport which im not subscribed to, if i load my roster i want to know the identity type so i can display icons that match

that ^

right, but treat it as stale and look it up at some point if you’ve loaded it from disk

of course, you have have a last seen attr and perodically request again

:+1:

all disco info is stale the second you received it, so this has nothing to do with application restarts

sure, but the longer you wait, the staler it gets :)

plus with caps+presence you will get push notifications if disco#info changes

flow, not for MUC services for example

and can then react on that and invalide your cache etc

which is what he’s talking about

flow we are talking specially about entitys that have no presence

I know. I just wanted to state that there is a difference between disco#info data with caps and that without

for those cases like MUC, smack has a cache which expires its entries after 24h

but actually, I am always pondering with that

assume we one day live in a word where the xmpp service address will host most services (which I think is desireable for the services where it is possible). and your service operator updates the service, then you will potentially only become aware of that new feature after 24h

stream:features to the rescue

it’d be nice to have 115/390-like push for services, too

wouldn't that be the case if you are subscribed to the presence?

for muc this somewhat exists

they just have to add xep115/390 metadata

its called notification of configuration change

i always disco after it

because the notification does not tell you what changed

flow, you’re assuming that all services expose presence

so it is probably not a issue of a spec filling a hole, but implementations just doing that

jonas’, well services need to know the subscriped entities to push 115/390 data to

that is true

I question whether that needs to be presence though

and I think there is nothing wrong with just re-using presence for that?

I can see the argumented of a polluted roster

I think there’s some reason in not using presence for this at all

but I wouldn't buy it

that, too

not for service-to-client, but for client-to-client presence, it quickly gets expensive to have all that stuff in the presence stanza

so I wonder whether more fine-grained notification models wouldn’t make more sense

hmm I'm sorry I can't follow, we where talking about services using caps to "push" diso#info to interested parties, and now you switch to c2s presences?

flow, yes

I’m questioning using presence for caps

because presence is overloaded

this is mostly relevant for c2s presences

(or, more specifically, for client-to-client presences)

so you basically want to re-open the old discussion between peter and phippo about presence floods? ;)

maybe :)

exactly

that’s why *not* having this in presence would be good

I mean clients do not change their disco#info often, do they? ✏

please elaborate, because I think having a client specific caps in presence is potentially the only thing sensible in presence these days

avatar hashes exist too

GPG also

well those should probably go in PEP

exactly

and openpgp is already in PEP

I still see it in presence stanzas

(if you use a modern XEP ;;)

just like avatar hashes

yeah, well

the question is, if we’ve gone through the effort to move this rarely-changing data out of <presence/>, shouldn’t we also move that other rarely-changing data (ecaps) out of presence?

what is the rationale for keeping it there?

ok, so from a protocl PoV we are fine (at least in these areas), seems to be more of an implementation-is-missing issue

jonas’, ahh, I was no thinking that the frequency of change should be a criterion here

I was more thinking of "is this client specific" as criterion

aha

I don't think you want different avatars for different clients

I was coming from the "rarely-changing data in a stanza which is often sent is a waste of bandwidth" angle

(course, if you ask enough people, some people will say they want this…)

yeah, the per-client-ness is an argument pro presence

though we’ve already had enough arguments for the case that per-client caps are rarely useful and most of the time you’ll need something like an aggregated caps over all clients of the user (both min and max aggregates)

and those caps could be distributed by the server in a non-presence vehicle and also contain full caps hashes for the individual clients which are currently online.

yes, but, even if per-client caps are rarely useful, which I do not know if I would aggree, I do not see this as argument to remove them

of course, what we have discussed regarding per-account caps appears still desireable

and we should move towards that

even if per-client caps are useful (which they sometimes are, I agree), the question is whether they belong in presence

and maybe, just maybe, we will discovered that per client caps are no longer needed, but then they will probably vanish automatically

yes, but I am not sure if this is the question we should answer right now

it appears as something deeply baked into the core of xmpp

not really, it’s in '115

that’s not that deep

and since they rarely change, i feel like it is not worth any effort getting rid of them

but if you want to work on a spec which puts those into something else (PEP?), then please do ✏

no, that they rarely change is a reason to move them out of presence

if they changed "sometimes", presence would be a good place. if they changed "often", presence would be a terrible placce.

if they change "rarely", they are dead weight in most of the presence broadcasts which happen

(of course, if they change "often", they cause unnecessary presence broadcasts, which is arguably much worse)

flow, yeah, I’ve been pondering that for '390, which is why I take the opportunity to discuss this

ahh you are not worried about caps triggering additional presence broadcasts, but the mere existence of caps in every presence ✏

exactly

tbh i never thought of this is something of a heavy burden ✏

it gets heavier when you introduce hash agility (like '390) and modern hash functions

I personally wouldn't invest time to improve here

stuff gets more and longer

true, but I do not think that we will change hashes often

I’m not so sure of that

sha1 has served us well for what? a decade or so?

and even if we don’t change them *often*, the transition period may well be a decade of sending two hashes with each presence

because oldstable

obviously

but if I had to guess i'd say we see 4-5 caps variants per presence at most

which is quite a lot

which surely is not optimal, but something I could sleep with

jonas’, if you really want to reduce wire bytes invent an XML extension where the end element is always empty ;)

ITYM EXI

alright this is getting WAY to close to what we discussed for OMEMO just minutes ago, I have to jump in with something slightly off-topic

We have the problem that for OMEMO, you subscribe to a PEP node of each of the contacts you want to encrypt with. And then we're flooded with PEP updates on each connect, because PEP sends an update automatically on connect (right?). We were thinking about compressing stuff with EXI 😀 ✏

Syndace, EXI needs to be done on the stream level

Syndace, a common pattern is to split PEP data into a hash and the actual data

it would be cool if PEP services could do that

that would solve the race condition issues around that

Nah, EXI is just compression for XML, not talking about the EXI XEP but the EXI technology

that way you only get the hashes on connect, which may already helps a lit

Syndace, EXI generates binary output though, I’m not sure you lose 99% of the advantages if you have to wrap it in base64 again.

Yeah that would actually help a lot

not sure if this is possible in your case though, would need to have a closer look

and if every client comes with an EXI implementation, we’re half way to being able to use EXI on c2s, which would be amazing

Did a bit of research on available EXI implementations. It doesn't look super good but there are open implementations for C, Java, JavaScript and C#, though I can't say anything about the quality of those. Seem maintained at least.

I hadn’t checked that far yet

I only checked if libxml supports it (which it doesn’t)

as much as like EXI, I am sceptic if this is the right solution to your problem

flow I think that should be possible for OMEMO device lists, I'll mention it as a possible solution, thanks 🙂

EXI could be used to compress bodies in general too, not only for the PEP node content

we should really discuss letting PEP services hash the contents of nodes

And we encrypt the bodies as binary data anyway (SCE), so we don't have to base64 stuff there

but that’d require c14n, and nobody wants to go near that in XML :)

Syndace, yes

Syndace, say them the OpenPGP XEP said hello (that is where the idea came from) ✏

...should really read the openpgp xep again 😀

jonas’> we should really discuss letting PEP services hash the contents of nodes +1

yeah that would be awesome

a generic optional mechanism where the push only contains "a hash" would be nice

could potentially be as easy as s/+notify/+notify-hash/

great, now I wanna write a new protoxep

when I actually wanted to go into the hammock

You can't do both? Write in the hammock?

flow, go ahead

> could potentially be as easy as s/+notify/+notify-hash/ damn that sounds soo good

+notify-hash-hash-hash-hash-hash-ha…

flow, why not versioning instead of hash

after all, a few hundred hash nodes that are the same as last connect also seeems kind of wasted...

larma, what would the advantage be?

jonas’, well if I have >100 contacts, each of them use(d) omemo,avatar,microblog,... and when connecting I +notify-hash, I still receive a few hundred hashes. And often enough those are just the same hash as last time I connected.

With some versioning scheme it could be done such that I only get the changes since last connect

larma, right

I thought versioning per node

where you wouldn’t win anything over hashes

versioning per contact or globally (by your local server) would win of course

true, yeah was considering global versioning

+notify-hash-$LASTHASH

😀

Uh, what have I missed here‽

Syndace, you do realize that that immediately causes a loop? :)

ah, no, not a loop

but still terrible things™

no I don't actually?

Syndace, that wouldn't work because what would you hash, after all you receive multiple different nodes based on that notify

Syndace, it’s part of your ecaps hash ;)

it doesn’t cause a loop -- that was a mistake on my side -- but it still does fun things. since everytime you receive a pep update, your ecaps hash would change and you’d have to reemit presence

oh right, you +notify for the node and not for each contact you want notifications from

jonas' I wasn't thinking about updating the hash during runtime, just setting it to the last hash you saw before disconnecting last time. Only to avoid the one single automatic update that you receive on connect.

hm are you aware that +notify-hash would change your disco info ✏

just saying that would flood me with disco info requests everytime something changes in pep

and this brings me to the topic that +notify is really bad in disco info

i cant deactivate a feature without changing my disco info

but on the other hand its really the most easy way to communicate to remote servers what you want hmm

"hash" means the word "hash" here, not any actual value. so you'd put "+notify-hash" in disco info exactly the same way you put "+notify" there now. so no disco info change everytime something changes in pep.

then i missed something

how does the server know on what version i am?

I don't think there even was consensus on doing versioning

just a simple hash of the node content

nothing more, nothing less

where is the hash of the node contetn

sent with the pep notification?

hash of what?

hash of the pep node content

what normalization?

what c14n?

whatever flow thinks of 😀

> sent with the pep notification? yeah. instead of getting the content in the notification, you get a hash of the content.

so instead of the actual data i get hundreds of pep notifications that contain a hash

reduced bandwidth and if you need to know the actual content you can manually query

and i have to query more if its not the hash that i have?

yup, instead of 100 device lists, 100 hashes

Isn't something like this in XEP-0060?

yes thats already in their

(for example)

notifications without payloads at least

its called omit the payload

how does that work with the first update you get when you connect to the server and +notify?

you get a notification just with the item-id without payload

the item-id could be your version or hash

cf xep84

but really thats not worth it for something like a device list on omemo

I've actually wondered why '84 doesn't just use payloadless notifications instead of a separate node

the payload is small anyway

lovetox well, the node can contain user-defined labels now

so it can be a few times bigger than in legacy omemo

for a few 100 contacts that adds up

at least larma said that the device list notifications already make up a considerable portion of the traffic on connect

It's probably not the only thing, but it's definitely visible

haven't actually calculated how much bytes it makes

you can reduce the payload

but this does not change the fact that pubsub in its current form is just not scaleable

it works nice until you reach X users in your roster

then it becomes a burden

you mean pep, not pubsub

pep is a subset of pubsub

payloadless notifications actually sound pretty cool, we'd have to set the item id to something with meaning though, like a hash of the content

we use "current" at the moment

then you need to configure the node to max-items=1

and do payloadless notifications work with PEP notifications?

which we sidestep with "current" right now

Syndace, its a node configuration, and you can configure the default node just to enable it

but this would probably break every client

heh 😀

sounds like something that might be a solution for OMEMO but I don't know enough about PEP/pubsub to push that idea forward

what we really would need is a smart idea how we can avoid notifications all together if we already received it

Syndace, you say solution like there is a problem

server-side device tracking?

pubsub-since?

omemo and all other pep based xeps work fine

its just not scaleable indefinitely

it doesn't have to tho, humans don't scale that well either

"problem" is a strong word, but e.g. we don't put the ik into the device list because it's too big, so you have to query the bundle manually for every new device.

openpgp puts the ik into the notification

and if everybody sets a huge label for all of their devices, you'll notice the traffic probably

so its not like it isnt already done

if the payload gets to big, you do what the other xeps do

add a metadata node

that tells you only the current version

see 0084

isn't that exactly what the hash approach would do?

yes, my example can be implemented and works tomorrow

yours need support in X server implementations first

and the result is the same, one is just more elegant

I think we're drifting away

why? its exactly what you want, you subscribe to a metdatanode, it always contains only the last hash or version

and you define in the xep, if the version or hash is outdated, you request the payload node

yeah sure, we talked about that for OMEMO

I don't know why we're reiterating it now

ok if im saying it now, i dont really see where the server could even help us here

The server could create the hash for us on-the-fly, without the need for an extra metadata node

but the extra node is on the server

for the client nothing changes

the client has to update the metadata node though

he gets a notification with the hash, and requests afterwards a node

when it publishes something

yeah true it has to publish 2 things

instead of one

hardley worth a new xep and serverimpl though if you ask me :D

its not like you publish daily devicelists

if you do it manually, every XEP has to do it manually. If you can just subscribe to #notify-hash, every client can decide to do it without the XEP even mentioning the possibility

> its not like you publish daily devicelists the problem is still the PEP update spam you get on connect

yeah true, as i said its a bit more elegant

yeah

Syndace, you also get a pep update spam with notify-hash

yes, but (in many cases) less :)

yes as it would be if you use a metadata node :)

less as in less bytes, not fewer updates

but ok, if the server does it for us it indeed elegant for the client

yeah. And it's easier than payloadless (is it?), because we can keep using one item with id "current" and don't have to rely on max-items (why not?).

the problem with payloadless is its a configuration on the node

so you have to get all servers to have this configured

which does not make much sense in other cases

`pubsub#deliver_payloads`

Hm, I wondered why something wasn't (also) a subscription option. Maybe this was it.

but the device list is its own node, isn't it? so you could just set that for the device list?

Did you not kill the 1+n node scheme?

Syndace, node configuration is not nice with client

we have two nodes, one with the device list and one with the bundles

first you have to pull the node configuration, then you have to set a new one

then you have to publish

the bundles used to be split into n

this is theroretically possible with publish_options, server support this only partly

single device id item or one per device?

single

two nodes in total

lovetox, easily solvable, just make the Conversations Compliance checker cry loudly about it

ah items, yeah one item with the list

Hm

lovetox we already require setting pubsub#max_items

so might also require the other thing

Zash, I think PEP only notifies about the newest item? That's why we want the whole list to be one item.

Another unshaved yak :/

Syndace, max items is supported by publish options on most servers

Why? 😀

other node configurations are not

If you could somehow ensure that you get all of the items, it'd be cleaner

(the Why was @Zash)

but yeah if the option is not set, its not bad

then you get the payload

And then you could use retractions to indicate device removal

Cleaner mapping

retractions are not sent on coming online

the one thing per item approach is good for stuff on your own server

https://xmpp.org/extensions/xep-0312.html uses relative time? 😱️

like bookmarks, which you want to request anyway on every start

Syndace, FYI https://wiki.xmpp.org/web/XEP-Remarks/XEP-0373:_OpenPGP_for_XMPP

so yes pubsub#deliver_payloads would be the way to go, the wiki page has a note about that feature being not discoverable though

cool! thanks for the link.

I think I had the split metadata and data scheme in mind becaue that is what works with any minimal PEP implementation

Like 84?

Zash, searching for devlier_payloads in 84 yields no results

it has split metadata and data tho

and since I don't have any detail of every protocol in mind, I would appreciate what exactly

aah ok

I also looked into into my notes and found a todo item regarding OK about deliver_payloads ✏

Syndace> payloadless notifications actually sound pretty cool, we'd have to set the item id to something with meaning though, Do you really have to set the ID explicitly? Often it is enough to go with the pubsub service generated one

soo good news, I don't have to write a protoxep, xmpp already provides what we need, we just have to implemented it in services and clients

and I can go in my hammock

Zash, actually I wonder if that split should be declared an anti pattern

before you consider something an anti pattern you should at least provide a different approach to reach the same goal

flow: Mmmm, borderline. I personally think the (old) OMEMO thing with 1+n nodes was worse. But if it works, it gets deployed.

flow actually there is a small but meaningful difference between +notify-hash and payloadless: payloadless has to be configured on the node while +notify-hash can be used on any node if the client wants to

+notify-payloadless would be amazing too

You could invent that

Would be easier if it was implemented as a subscription option tho :/

and specced as one

and payloadless should probably be made optional with a disco feature to reflect the current state of server implementations

Is there a feature for it?

> the feature is not discoverable, most likely because it appears to be mandatory by XEP-0060

from https://wiki.xmpp.org/web/XEP-Remarks/XEP-0373:_OpenPGP_for_XMPP

It there a feature for `pubsub#deliver_payloads` I mean

if https://xmpp.org/registrar/disco-features.html is the list of features then no, can't find anything for "deliver_payloads"

Anyway, the situation is quite clear, we can't rely on any of that for OMEMO. If we want to reduce the on-connect PEP update traffic, we have to manually specify some sort of metadata node.

Account level subscriptions + MAM? :)

Any I think we agreed that it's not worth the effort given that the device list node is rather small generally

I don't think a hard dependency on MAM is a good idea just for that

how does account level subscription work? you subscribe using '60 instead of +notify and then you receive updates as messages that are stored in MAM while you're offline?

Syndace, you subscribe your account, notifications are sent there and could /in theory/ be forked (instead of the origin sending to each +notify) and MAM'd

In practice those notifications will just be discarded because type=headline

ah, right

Could be solved by some future magic XEP probably

IM NG might help actually

Also possible to configure notifications to be sent as type=chat, but that's a node config, not subscription option :(

meh

More of these fancy things as subscription options would be awesome

So each subscriber decides

Syndace, yes, but do most xeps, including OMEMO not already specify how nodes should be configured? so I am not sure about how meaningful the difference is in this case

I think we should probably tackle this from two angles: configuring the node to not deliver payloads *and* invent +notify-payloadless

flow, how would you introduce +notify-payloadless to a federated network?

larma, haha .. :(

well the problem is that it's not relying on your server to be updated, but on every server to be updated

larma, you do both +notify and +notify-payloadless and the receiver needs to ignore the former?

larma, ^

So as a client I get mixed responses, sometimes payloadless and soemtimes not?

depending on what the other ends servers supported

well ideally the node is also configured to no deliver payloads

I actually think that this should be enough

Alternatively, mandate local server magic

Your server could easily hash the content (/me laughs in xml c14n) and forward you the hash

Zash, I don't think c14n is relevant here

If we do local server magic, I'd rather go full server magic and do global pep versioning

Oh

I'm confusing the hash stuff with the payloadless stuff

nm me

Should be trivial for a server to strip the payload

even there, do not think of it as a hash, but an Abstract (PubSub) Node Data ID

for which is true that if the node data changes that abstract id changes to

I thought I saw some talk about hashing the payload data somehow

but it is not important that "similar" xml leads to the same id

infact, if the data does not change, but there is a new id, that would be fine too

one could even implement that abstract (PubSub) Node Data ID as counter

i.e. it is the same id that xep395 would use

Anyways, a server seeing a pubsub notification that includes a payload, but the client has set +notify-payloadless then it should be easy to strip the payload and forward the rest

IIRC payloadless notifications still include the id, so if you stick a magic hash there you should be golden

why is sticking a magic hash in there important?

couldn't you just use the, you know, item id?

flow, if item id is just 'current' all the time, that's not very helpful

larma, it has not to be that way

yes

isn't 'current' just because singleton node? ✏

yeah, but then taking a hash instead of a random number is a good idea, because changing back and forth will result in same id so no unnecessary requests for those that were not online in between

now the question is if it is possible to keep the singleton semantic but have differnt IDs for different items, which seems desireable anyway

yes, but you need max_items

larma, it is a good idea without doubt, but it is not strictly required

Zash, and we do not have max_items? or is there no issue?

I think we do, but there's some extra care involved

Older prosody doesn't, but also doesn't support >1 items so it's fine.

so, use max_items=1, delivery_payloads=false, service generated item id, → $$$

can't we just build mam for pubsub instead, maybe using some shorthand +notify-archive which will cause the server to automatically subscribe to nodes and deliver updates from the archive when connecting?

🥇️

larma, any question including the word "just" automatically get "no" for an answer

It's never "just" anything :P

😀

life would be boring if things where that easy

It's not about doing something that's easy

It's rather about doing something that's meaningful

larma, pubsub+mam has been mentioned in the past as the glorious saviour of everything, but it's lacking in actual specification or somesuch

lacking in "how should it even work?"

I think MattJ had some stuff to say about it recently

replacing `<item id='current'><devices><device id='1' /><device id='2' /></devices></item>` with `<item id='b5fec669aca110f1505934b1b08ce2351072d16b' />` isn't really a huge improvement IMO

How many phones and computers do normal people even have?

Sure it's some improvement, but it still means O(n*m) traffic on connect (n = number of contacts, m = number of features that use pep)

I alsways calculate with 3, but I feel it's probably rather 1.7 or something

many don't even use their notebooks/desktops for messaging at all

I heard computer sales was picking up because everyone needed to work from home :)

"I was sending SMS from my phone for the last 25 years, I'll continue to do so"

thats what i said earlier, the whole idea makes it a bit more efficient, but in the whole great scheme of things, where everybody stuffs all into pep, it does not really matter

but nevertheless it would be more elegant, and xeps would not need to define metadata nodes anymore

lovetox, have you heard of https://en.wikipedia.org/wiki/Jevons_paradox ? :)

no i did not, but now i know

should not forget in the size comparison that there are labels

and clients will probably set default labels of a few chars

Labels?

optional labels (=strings) for devices

to make it easier to identify keys

e.g. Gajim could set "Gajim on Windows" for its default label