XSF Discussion - 2021-07-20

ben, ben:

Hi, can anybody confirmS my understanding of https://xmpp.org/extensions/xep-0060.html#owner-purge: "If a service persists published items, a node owner may want to purge the node of all published items (thus removing all items from the persistent store, with the exception of the last published item, which MAY be cached)." . So the last item (as in most recently published) must be kept right? Because on one hand we have "purge the node of all published items" and on the other "with the exception of the last published item, which MAY be cached". (ralphm ?)

Sounds like it imagines a tiered storage system, where you have the persistence layer that stores `max_items`, and then a cache of the last item for re-sending on subscribe or presence. Purge would primarily operate on the persistence layer, but why not both?

and the caching is an implementation detail anyway. For me purge should remove all items from the node, not keeping the last one.

That seems like the sensible way, yes. But the spec allows keeping it 🤷️

I can deal with that, but I'm not sure about the spec, it just allows to keep it, or we HAVE to keep the last one? I hope it's the later, because otherwise when we have a purge event, we don't know if the last one is still there or not.

"MAY"

https://datatracker.ietf.org/doc/html/rfc2119#section-5

> mean that an item is truly optional

I understood it as the MAY is for "cached", "MAY be cached".

so the way I get it, is it MAY be cached, but we remove all but last item.

Hmmmm

I read it as, if you need to keep the last published item, e.g. for PEP use cases, a service MAY keep the last item when a node is purged.

Also, I don't remember this changing. I agree it is confusing.

OK, so we don't really know if last item it kept or not when we receive the purge event?

Speaking of pubsub, I wonder if I've misunderstood the whole persistence thing at some point. In Prosody, the `persist_items` setting toggles between persistence to storage and persistence to an in-memory cache. So `persist_items=false` just means that data is forgotten on reboot, but otherwise it looks the same from the outside.

Now I'm wondering if it's actually supposed to forget stuff after broadcast. And then maybe a different cache with 1 item for rebroadcast of last item.

Zash I understood it as the former too, but the later would make sense, and after checking the spec, it's not really sure which one it is.

Yeah, I'm not sure whether the items are stored in memory or on disk is really useful (or should be controlled by) a client

Time for Pubsub 2.0 :)

Before or after The Split happens?

Zash: yes, that's wrong indeed. With persist_items false, you don't store them in any way and also if the item doesn't have an id you don't have to generate one for the notifications.

The method of storage, or any other type of guarantees on the retention cannot currently be controlled by the client in a standard way. Other than maybe the pubsub#item_expire config option.

goffi: to be honest, I think the exception for the last item feels weird to me and I wouldn't object to removing that.

ralphm: it would be possible on a draft XEP ?

Let's see what the council thinks after you PR the change :D

Don't forget to discuss it on the standards list :)

This author doesn't object, if that helps.

ralphm: I'm finishing a protoXEP to indicate extra data giving details on Pubsub implementation, so we can safely cache a node, I'll propose it later this week. I've added this case, so maybe it's not necessary.

s/MAY/SHOULD NOT/ ought to be safe from a interop perspective, but then who'd rely on a cache anyway?

I'm not even sure what the difference is between caching and storage, from a protocol perspective.

Indeed

ralphm: I mean for caching a pubsub service from a client (local cache)

but I guess I'll propose the protoXEP first, we can discuss after.

Unfortunately when Kev converted the repo from Subversion, we didn't keep the older history, so I can't check how it got in.

In any case somewhere between 1.6 and 1.9

goffi: by the way, you *can* detect this. There's a `cache-last-item` feature. Apparently it doesn't appear in the list of features in section 10 nor is it registered with the Registrar.

That feels like a bug

ralphm: one of the problems is that those features may only be accessible to node owner (even to only read the value).

wait huh?

those features are in node configuration right? Node configuration may not be accessible from a random user.

Why not? They should be available via disco

If they are not, all bets are off. You wouldn't even know if a node persists items in the first place. Also, the purge action is for the node owner, who should definitely be able to see the node config.

ralphm, its not a give that every configuration is avaliable as disco info

we had similar issues with the MUC XEP

Yeah, those things are ugly :)

but this is i guess in part because it was never intended for that or ?

its about what features a service supports

not neccessarily how the service is configured currently

Yeah, I think it probably began as "it's useful to see some of the important configuration info" + "we don't want to dump the entire config into disco"

and they don't necessarily map 1->1

e.g. config option 'roomconfig_whois' maps to the features 'muc_nonanonymous'/'muc_semianonymous'

I don't think it's a bad design given those constraints, but it could do with a bit more consistency

lovetox: I meant node meta data discovery (section 5.4), where the node's configuration is passed as a Data Form in the disco#info response.

This is the only true way to find out if, say, a node supports persistent storage. I know the section is a MAY, but not supporting this is meh.

ralphm: it can be configured per node, and it's not in node disco (at least it's not enforced). That's exactly what my XEP is doing: making sure it's in metadata. Also purge is done by node owner, but if I cache a node as simple subscriber member, I want to be aware of the purge.

You'd get a purge notification?

if I'm a subscriber, yes

why would not I get it?

That's the deal, I guess. If you are a subscriber, you get notifications.

yes, but not all. There are lot of corner case. For the purge I get a "purge" notification, but if I don't know if the last item is kept or not, I can't replicate the command in my local cache.

Sure, I understand that. I agree that implementations should expose node meta data through disco. FWIW, an implementation that does should present that in a regular disco#info feature. Also this feature is RECOMMENDED.

What silly implementations don't support this, really?

(pubsub#metadata, I mean)

I haven't seen it a lot in the wild to be honest.

did you mean `pubsub#meta-data` :D

No

Zash: oh, crap. Well, I guess we can add a ticket against Prosody :-(

and ejabberd

and propably most pubsub implementations

goffi: I mean that Prosody presents this with http://jabber.org/protocol/pubsub#meta-data instead of #metadata.

oh prosody does it, OK.

Oh, cool. Even idavoll has #meta-data. /me hides in shame

I can't tell if ejabberd announces support for Node Metadata, but it does implement it as far as I can see.

yes, and non owner can't get config

how it does sorry, I've misread

it does? I have nothing on a ejabberd PEP node

https://github.com/processone/ejabberd/issues/1421

Although I wouldn't be surprised if it behaves differently for PEP nodes.

weird, I'm actually testing on edhelas blog node I don't see any metadata.

Huh

oh

How did this happen?

😱

Zash: don't worry, if all actual implementations use #meta-data, we can change it in XEP-0060 :D

what are you looking for ?

edhelas: Pubsub Node Metadata (see section 5.4)

I see

``` <field type='hidden' var='FORM_TYPE'> <value>http://jabber.org/protocol/pubsub#meta-data</value> </field> ```

Zash: well, yes. But that is kinda useless. It should be in a <feature/>, cause otherwise, how'd you know to expect the form?

oh never mind

It looks like tigase has http://jabber.org/protocol/pubsub#metadata.

And basically anyone else: http://jabber.org/protocol/pubsub#meta-data

alright I see it on a non PEP node with ejabberd (with meta-data :-/)

but not in the PEP node

goffi: happy for you to push ejabberd to start supporting this in PEP nodes.

And no more info on the change that added this than "mod_pubsub: Advertise title and description in disco#info"

Zash: I'm not surprised. Idavoll was one of the first implementations, so I wouldn't be surprised if that had a knock-on effect.

So no idea if _someone_ copypasted from ejabberd, or from an example shown by whoever requested this, whatever it was

well 2 wolves found with Pubsub specs/implementations (not sure if this expression translates to English, but whatever), we have not lost our day.

Write longer commit messages plz!

(it was me)

My commit to add support was on 2005-01-02.

this was just 2018

The version of XEP-0060 supporting this was published on 2004-07-13.

(the disco feature)

Heh, nodes only have a leaf identity and a form in disco#info

hm

Someone pointed out recently that disco#info itself is required (*hint* if someone wanna send a trivial patch)

And the service, because that's where the feature should be published

Whatabout individual nodes?

That disco#info thing makes no sense to me :)

Mmmm, yeah, getting a non-error response kinda implies that it's supported :)

Zash: I think nodes should only have the http://jabber.org/protocol/pubsub feature

But I'm sure we can clarify if nodes should have disco features other than that.

I _think_ the features listed in Section 10 are service wide, though.

Also, I don't there's anything particularly wrong with announcing all the features on each node, but not needed.

Did you already notice that there's no #meta(-?)data feature? Only the form

Zash: in Prosody you mean?

In XEP-0060

Orrr, wait

https://xmpp.org/extensions/xep-0060.html#registrar-features does list it

Section 10 does, too.

Ah, because I searched for "pubsub#meta"

But then the FORM_TYPE is a bit like a feature if you squint at it

Oh, there, now I see pubsub#meta-data on the service itself

yay

except for the dash, of course

I'm not sure how to treat this, spec-wise. Curious if there are other implementations that have #metadata, besides Tigase.

intosi: what does MLink have?

What do clients expect?

M-Link doesn't do PubSub meta-data.

At all?

At all.

Didn't expect that, but at least it doesn't use the wrong namespace then.

Zash: Clients might not actually check

WHAT

https://xmpp.org/extensions/attic/xep-0060-1.13.2.html#entity-metadata here it is #meta-data ??

https://github.com/xsf/xeps/commit/660ff16274cebad556115f94b6ce7550c17b32e5 https://github.com/xsf/xeps/commit/37c7b852f302fca26a885e9d2c2aea5de019fc3e https://github.com/xsf/xeps/commit/6842676930122aaa2988cf3595175849fd7fa4f8

Aha

jonas’, editors, something for you to look at? ↑

OUCH!

pep. strikes with search/replace :-D

Also https://tdan.com/the-meta-data-fiasco/18502

why the rerevert though

I recall the issue, but not the resolution

It's very nice when you track down an issue and find a commit with a long commit message that explains the reasoning.

I'm not sure the assertion that "meta-data" being wrong is true.

ralphm Michael Brackett has a wrong understanding on what a tautology is. "metadata" (= data about data) is not a tautology, a "discussion about discussions" is not a tautology etc.

Sure, but the bit about trademarks is interesting

I also don't think that linguistically, putting a hyphen into a compound word is wrong.

ralphm I agree, that the hyphen were wrong in English, but we Germans *love* to put wrong hyphens in English compound nouns!

Well, I know that in Dutch, compound words generally cannot have a space in them, and hyphens can be inserted for clarity.

If I were to talk about "Service Discovery" as a concept in Dutch, borrowing the English term, I would have to write "service-discovery".

Simply because "service" is not (used as) an adjective.

(and Dutch also doesn't capitalize nouns like German does, so you can't tell the difference)

jonas’: shall I put in a PR?

> ralphm I agree, that the hyphen were wrong in English, but we Germans *love* to put wrong hyphens in English compound nouns! Simply concatenating two words is certainly wrong in English, though. (Except when it isn't.)

I find it convincing in Bracketts text, that "meta" is a prefix word and therefore does not have a hyphen, so "metadata" is correct in the prosa. However, the XML namespace is a different thing.

https://metalhead.club/@holger/106133744976748781

Like Bracketts example "metaphysics" etc.

Something something "sär skrivning"

Holger: well, hyphenated compound words tend to become closed compound words with time. E.g. if a compound word is used "enough" the hyphen tends to be dropped eventually.

the heck

https://github.com/xsf/xeps/pull/710

so either this was a typo in some commit id or I don't know

I apparently merged it without re-checking closely...

ralphm, go ahead please!

ok

Kev, repoke: https://github.com/xsf/xeps/pull/1047

also, sorry everyone ;)

jonas’: for what? The meta-data thing? Maybe you are 'just' human?

Errare humanum est.

FYI I've just proposed a protoXEP to fix some issues mentioned earlier today: https://xmpp.org/extensions/inbox/pubsub-caching-hints.html .

jonas’: https://github.com/xsf/xeps/pull/1092

ralphm, thanks!

You're welcome. Zash: thanks for the find!

np

Also put in a drive-by comment over at Tigase.

Look like they only put that in in May this year: https://github.com/tigase/tigase-pubsub/pull/5

Looks

The addition in Prosody was only a few months before, ejabberd seems to have had it a couple years more.

jonas’, that was a lot of emails! (thanks for the work :D)

well, automation did the job

thanks to Sam for pointing out that I should trigger the automation more often :)

yay scripting

pushing a button to run a script is still work

It sure is. This is what pays bills (in other places)

Wojtek: fast turn-around :-D

owning to Andrzej :-)

We don't have a standard way to indicate in disco that a feature is implemented for a something but not something, do we? For instance on a server, we want to indicate if RSM support is for PEP/Pubsub and/or MAM and/or disco items. There is https://xmpp.org/extensions/xep-0060.html#subscriber-retrieve-returnsome with its "http://jabber.org/protocol/pubsub#rsm" but I have the feeling that we need something more flexible, this looks like a piece of sticky tape.

something but not something else*

goffi, no, cross products of features are an issue. the similar is similar with SCE: if someone announced support for SCE and, say, chatstates, does that mean that SCE-encrypted chatstates are also supported?

Personally I would lean to the safe side and usually ask for an explicit announcement of those things

Because I'm updating the Order-By XEP following feedbacks, and I need a way to indicate if it applies to Pubsub, MAM, or anything else.

What do you think of `urn:xmpp:order-by:0#urn:xmpp:mam:2` (so `feature_supported#where_it_applies`)? Do you see any problem with this kind of notation? It would be flexible and I could publish a new XEP to formalise it.

maybe something else that "#" as "#" is used by some protocols to indicate sub-features (e.g. pubsub)

or two "##"? I can try with that and see what happens on standard@

goffi, hmm, what about urn:xmpp:order-by:0@urn:xmpp:mam:2

but, tbh, it all looks rather ugly

if you have a better options, I'll be happy

otoh, something like this would also been my first attempt at this :)

If have against "@" and it make sens as it means "at", I just find it slighly more difficult to read ("@" is close to "0"), but that's a question fo taste, not really relevant.

I have nothing against "@"*

makes sense*

Why do we actually not have a nice XEP bot here? ☺ Alex: anything possible? 😃

In the old days we had one. But I can't remember who wrote and maintained it.

emus, what would it do?

point you to the xep on request or other details

on which request?

what I saw today was: !xep XEP Number -- recieve information about the specified XEP

emus, why a bot when you can just ask Link Mauve ;)

😀 work that a machine can do should be done by a machine 🙂

Is it a coincidence that Link Mauve means Purple Link, which is what followed links are usually colored as?

uh, wouldn't they be Lien Mauve?

Alex, jonas' https://github.com/mightyBroccoli/xmpp-chatbot