XSF Discussion - 2018-02-19

anyone here at whom I can bounce an idea I had for ecaps2?

(otherwise I’ll take that to the list, but from experience, I won’t get a lot of feedback there :/)

How much context is needed?

if you know how caps (XEP-0115) works, it’s probably fine

How much coffee is needed? :)

uh, maybe a bit, but I’m not sure. I don’t drink coffee :-)

Shoot. What's the worst that can happen? :)

hah ✏

so I was wondering what if we don’t have clients inject the caps into the presence, but instead use a nonza/stream-level element to let the server know about the caps.

the server would then re-send presence and take care of injecting caps updates in presences.

receiving entities would see the caps in presence as usual.

this saves the client from re-sending (non-directed) presence when the caps change (server would re-broadcast)

I literally just had the idea. I’m not 100% sure how the interaction with stream management would be.

(but probably it would be "re-send your caps to the server after resumption, unless you’re 100% sure your previous caps update came through")

the rationale why this might be nicer for the client is that presence is already a rather complex beast, in the context of MUC, in the context of vcard avatars and possibly other things (if we want to integrate chat states into that for example)

You could transmit your entire disco#info blob, then the server could do the caps/ecaps/magic for you

very interesting point

and do all other sorts of interesting things with it even

(e.g. what people proposed with publishing a subset/superset of all client features in a PEP node)

Would probably tie in nicely with some future client registration

client registration?

like, device keys?

The server having some idea of which clients you have.

yes

Sounded like people wanted something in that direction.

jonasw: I think we should be careful with just throwing non-stanzas at everything

an IQ would work just as well here

(I actually put some thought into nonza vs. message for server-originated pushes and there nonza was fine and semantically made more sense, so I just transferred that to client-originated pushes too; I might be wrong here)

(I’m not sold on using nonzas)

Caps are a property of what, exactly?

the client.

(or server respectively)

(entity.)

Not the stream itself.

yeah

probably makes sense to put them as IQs or something

I was thinking basically just disco#info in an iq-set would be nice and simple

yeah

so clients wouldn’t have to calculate *their own* hashes anymore, but they might still want to calculate other peoples hashes and keep a cache.

for directed presence, sending an "empty" directed presence would make the server inject caps as needed. is that a privacy issue?

probably not, because who knows your resource can disco#info you anyways.

yup

cool

I hear a namespace bump rolling along.

or maybe not.

gotta think about that

New namespace, disco#push or something?

nah, I need to think whether caps2 itself needs a bump for this. But it might actually not.

Does ecaps2 replace disco?

no

not entirely I think

caps is a separate thing right

it might save you a lot of disco#info-ing, but you still need to support xep-0030

and then I’ll need to make an implementation in prosody to show that it works :D

PoC all the things!

maybe I’ll sink my afternoon in that.

jonasw: awesome idea, I'm all for making the client dumber again

jonasw: beware directed MUC presences though, they are broken already.

Ge0rG, what’dya mean?

server wouldn’t resend directed automatically, client needs to do that (server just injects caps hashes)

jonasw: I _think_ I sent that to standards recently, but can't find. When the MUC changes your nickname, your server will later send your "directed" unavailable presence to the old nickname.

yeah, heard about that

jonasw: shouldn't the server resend all directed presences on a caps change?

Yeah, the MUC changing your nickname causes all sorts of headaches

Ge0rG, I looked at how {xep 0398} does it, and it doesn’t resend directed either

Ge0rG: User Avatar to vCard-Based Avatars Conversion (Standards Track, Experimental, 2018-01-25) See: https://xmpp.org/extensions/xep-0398.html

jonasw: maybe it's just an oversight by the XEP authors?

it would make sense to resend.

maybe; we should ask daniel about this ^ ✏

jonasw: Nothing says you can't have the server query disco#info like normal and then generate caps and ecaps2 for it.

For legacy clients etc

Zash, true; I’m not sure why you’re mentioning that though.

Thinking out loud

(and it would require polling)

ok

it would add some nastyness into the protocol flow

Prosody already does it, but discards the response, save for the PEP +notify items

1. client sends online presence 2. server delays outbound presence, asks client for disco#info instead 3. the client sends directed presences all around 4. eventually, the client responds to disco#info 5. the server can now inject and forward

is +notify part of disco#info?

Ge0rG: Yes

Zash: which question was that a response to?

I wonder if we should be stripping +notify from things sent to peers while we’re at it.

Ge0rG: +notify goes in disco#info

but probably not.

Can we forklift PEP into being account based that way?

As in, subs are between accounts, not between remote account and specific resources

wasn't there a discussion of making PEP account-based just some days ago, on this MUC?

I must’ve missed it

Zash, it could be a good first step with the superset thing

would probably require an update to {xep 0163}

jonasw: Personal Eventing Protocol (Standards Track, Draft, 2010-07-12) See: https://xmpp.org/extensions/xep-0163.html

Zash: what's up with Bunneh? I'm missing it sorely.

Bunneh: ping

Zash: pong

Bunneh: version Ge0rG

Zash: Ge0rG is running yaxim version 0.9.2-95-gcab21a3 on Android

That's a lie.

Ge0rG, your statement is confusing to me: https://sotecware.net/images/dont-puush-me/v7RTW5gsnGjnTE5G6QahCIPsruzqV3-1IRISifpDBRI.png

Ge0rG: Stanza reply id duplication causing you problems?

Zash: it won't join my MUC.

jonasw: why are you quoting by image? :P

Ge0rG, that is a very good question I can’t really answer

I guess that’s what happens when you have a scrot-scp-xclip pipeline bound to something which is close to a hotkey

I don't even want to know what "scrot" is the abbreviation for.

Ge0rG, it’s not an abbreviation, it’s the name of a tool.

I find it confusing, too.

that's what she said?

cf. http://manpages.ubuntu.com/manpages/precise/man1/scrot.1.html

argh

I’ll just head out now :P

Ge0rG: I'm going to guess that it thinks that it is joined. Or doesn't support mediated invites. Or both.

see you later

Zash: can't you kick its lazy bottocks?

I can update its server

If that will make it rejoin my MUC?

Setting up prosody-trunk (1nightly844-1~trusty) ... prosody stop/waiting prosody start/running, process 1714

When doing disco#info on my own server, how am I supposed to differentiate a MUC service from an IRC transport?

Are you supposed to at all?

I'm sure it won't end up well if I try to create a private MUC on biboumi.

Uh, fun

Wasn't there a disco#info item telling a client that it may create new MUCs on that service?

I'm not aware of such a feature

Not that many features depend on who asks

Or?

...that it is possible to create new MUCs on this service.

XMPP is full of fail.

You could look for the feature "http://jabber.org/protocol/muc#unique"

It's pretty useless tho

It would imply that it's possible to create randomly named MUCs... for someone... maybe

Zash: that's not part of 45?

-xep 307

Zash: Unique Room Names for Multi-User Chat (Standards Track, Deferred, 2011-11-10) See: https://xmpp.org/extensions/xep-0307.html

It was broken out IIRC

The Prosody implementation just returns an UUIDv4

Ah, it's type=irc as opposed to type=text. Yay.

How is irc not text?

And how do you know that room creation is not restricted to admins?

I can't.

I can't fix all problems of XMPP. Not today.

It's all an iconian plot I tell you. 🙄🖖

Ge0rG, but you have to

mathieui: maybe if somebody will pay me.

Ge0rG: do you accepty payment in the form of stickers?

Guus: I'll ask my landlord.

TBH I never considered putting stickers onto my Laptop, my car or any other item. Not sure why.

Ge0rG: Cut out the middle man! Stickers make for nice decoration for inside a cartboard box under a bridge!

Ge0rG: same here.

Guus: great idea! But living under the bridge isn't free either.

Extract tolls from passing goats?

I could probably obtain sufficient amounts of food by dumpster diving. But how should I handle the -5°C temperatures

Ge0rG, mine bitcoin on your laptop

oh wait.

I like how we as a community are coming together to tackle both Ge0rG's residential issues as well as all of XMPPs problems.

Guus: I'm very grateful for that effort.

,oO( does "being grateful" actually mean "being full of grates"? )

Full of gratitude, I think, but your understanding could be good too.

In that case I prefer to be gratitudeful, thanks.

Hmm, gratäng?

jonasw, I'm not sure if client implementations would become easier. As far as I remember, implementing caps in Smack was not really challenging.

besides the i;octet collation, which is possilby underspecified in xep115 (but thankefully not in xep390)

I would be happy if, when I use `clix raw` and send presence, it doesn't instantly drown in so many disco#info queries and other stuff that the terminal crashes.

(rlwrap doesn't always handle syntax highlighting all that well)

Are there so many implementations that fetch disco#info unconditionally after a caps update? Are impls supposed to do that?

Zash, what's clix?

Zash: thats separate though, we can have query interception as-is

Command-line XMPP.

But yes, I'd expect clients to disco when they see a new caps.

Why?

flow: http://code.matthewwild.co.uk/clix/

To know what features are supported by contacts.

flow: decoupling caps from presence has the advantage of being able to use caps without presence

jonasw, is it really caps that you decouple, not disco#info?

“15:54:05 flow> jonasw, what do you think about the xep390 optimization to move the disco#info result from the client to the server?”, I have an experimental module doing that for 0115, caching the disco#info and the 0115 node in the session table, and answering iqs requesting it automatically.

and if we push disco#info and let the server deal with it, thats nice I think

A few things to fix and it should end up in prosody-modules.

also I'm on mobile

gotta go

Kev, but why do you need to know the features right after the caps update? And not, let's say, when you show the contact list in the GUI?

Link Mauve, so xep115 § 8.4?

flow, hmm, no, I don’t change anything about what the client advertise, I just request the disco#info when its <c/> changes and answer iqs from external entities instead of routing them to the user.

I’m not sure what stripping it would achieve.

Other than increase the amount of requests from other entities.

And prevent them from using their normal cache mechanism.

Link Mauve, sorry, I thought that was the xep115 counterpart of xep390 § 6.4 without closing reading it

So you do xep390 § 6.4 for xep115?

flow, yes, except I forgot the third point, will fix.

I’ll add a mention of that.

And soon will add support for 0390 too.

Link Mauve: sweet! what do you think of the proposed model of pushing disco#info from clients?

so that the server handles hash calculation and presence broadcast?

jonasw, would avoid one roundtrip, but it would also require every client to change, so I’m not sure it’s really useful.

Well, both clients and servers.

Link Mauve: we need to change clients for 390 anyways 😺

(and servers)

Uh, why?

I just had the idea to let the server compute 0390 and insert it in the presence if the client only had 0115.

Link Mauve: because they need to emit/process 390 hashes instead of / in addition to 115 hashes.

That would immediately increase adoption and usefulness.

yeah

indeed

I'm working on an update to 390 btw. I'll add that.

So I’ll continue this Prosody module in that direction, and you fix all of the clients in the meantime. :)

I won't fix pidgin 😼

jonasw: but you could burn it with fire.

Ge0rG: having the server re-send directed presence to MUCs also has interesting isszes with the proposed no-join (since errors need to be handled) and possibly passwords etc.

I think thats a can of worms I don't want to open.

gotta go agaib

*again

re

Link Mauve, so, others (I think the MIX folks) have already indicated interest in pre-presence disco#info from the client. XEP-0390 actually already specifies that with an IQ (even though that IQ only carries caps instead of disco#info). the change would be to make clients always use the IQ instead of presence to send caps to their server (and the server takes care of injecting caps in presence and re-broadcasting non-directed presence when caps change)

Zash proposed to change the IQ to actually contain the whole disco#info data instead of only the caps, which I think is neat. It allows the server to do all kinds of interesting things, like do the 115+390 optimization without having to ask the client for disco#info if it doesn’t have the hashes in the cache.

Wasn't there a proposal for rolling auth, resource binding and some session initialization into one transaction? Maybe something Kev or Dave talked about

https://opkode.com/blog/xmpp-chat-badge/ this is neat (from jcbrand)

Now taking bets on how long it would take to make that as a prosody module

Bunneh¡! 🤣

ah, we have that for user presence, but not mucs. That's a nice idea

I shall shamelessly copy that.

References to pubsub items. Thoughts?

We're about to allow references to FDP forms in Swift, so we want a reference to a pubub item.

Kev: I would like to reference pubsub items for mentions in blog post too

Kev: I though a super simplified xpath compatible syntax would be useful

+t

We could just do this as any other URI reference, pointing at the pubsub item, but if we did a specialist 'pubsub' reference we could also include the namespace.

Kev: URI is not enough, we need to know if we reference text or XHTML body

I don't follow.

goffi: for pubsub?

intosi: for microblog items

But that's pointing to stuffs inside a pubsub item payload, which is probably beyond what Kev;s intending here.

I'm talking about just pointing to a pubsub item.

intosi: yes it's pointing inside payload, that's why I'm mentioning simplified xpath

OK

so that's an other issue I'm talking about then.

I think so, yes.

Speaking of xpath, does anyone do some form of stanza matching that doesn't involve xpath? If so, what do you do? I've been struggling for ages to come up with a way to match handlers in an XMPP library that I don't end up despising and throwing out.

SamWhited: Seen Prosody's approach?

waqas: I must have used it when writing plugins, but I can't for the life of me think what it is?

For each stanza, we fire some events on our event bus. Event names like "iq", "iq/{xmlns:tagname}", etc.

Because Prosody is a server, we actually do "iq/{bare|full|self|host}/…", etc, since the target JID's nature actually matters for server routing.

ah yes, I remember. What if, for example, multiple things listen for an IQ and both try to reply to it?

iq/{host,bare,full}, iq-{get,set}/{host,bare,full}/xmlns:name

Our event bus is a priority queue, and the first handler to indicate it has handled stops the event chain.

*nods* that makes good sense

Handlers can be registered with priority, so e.g., a privacy list handler may want to have a higher priority than the actual routing. Default stuff is priority 0.

I actually had something similar to that a while back (except it was chains of function calls similar to the HTTP middleware pattern, and any of them could short circuit the pattern). I can't remember why I didn't actually end up using it now though…

If multiple things register with the same priority what ends up happening?

Random order basically.

I think the reason I didn't do something similar is that I needed to match initially on the IQ start element token to route to the IQ handler chain, but then in there I needed to match on the payload, but I didn't have a good way to pass both that token and the stanza start token on.

But maybe I should give up and just always decode the entire IQ as soon as I receive the start token and pass a representation of the entire thing down through the chain

We just have mod_iq hook the top-level IQ event, and it then fires the sub-event :)

FWIW, aioxmpp lets you match IQs requests by payload (only namespace+localname essentially) and get vs. set

Prosody would parse the entire stanzas before fireing events for them

that, too

Full xpath is overkill for XMPP routing

yes

At least in a server

I suppose I'll have to do that, right now my handlers look something like: HandleXMPP(xmlstream.TokenReadWriter, *xml.StartElement)

SamWhited: Are you trying to avoid parsing into a DOM immediately?

for things like pubsub where you’d want to match on deeper things, the pubsub service actually eats all pubsub-related things from the stream and emits its own events

waqas: yah, it's a tad bit more expensive so I was hoping to do the match and decide if I need to do it or not first, but I'm starting to think there's no good way to do that

https://instantdomainsearch.com/articles/streaming-json-jsons/

SamWhited: I'd advise against it. The very minor speedup isn't worth the increased code complexity and mental overhead, except for very selected pieces of software where you must have every fraction of perf.

And you are parsing the whole thing anyway, the DOM cost isn't actually much (though non-zero, hence my experiments with in-place alloc-free DOM construction).

Yah, I say "parsing" but that's not actually the expensive part since I have to do that anyways, it's allocating a place to store some generic representation of it that's expensive.

I actually don't have a decent way to represent a generic blob of XML either, but that's not the end of the world, I've written one for the (terrible) xml library I'm using, so I could probably copy/paste that until it gets merged upstream.

What I mean is that that allocation is a fairly small fraction of the allocation that's happening anyway in your XML parser. Every element name, attribute name, attribute value, etc is causing at least one allocation.

It's true; I like the idea of being able to swap my inefficient XML library out for something a bit better later though and not have my code be the bottleneck

Just give in and refactor later. Bets on it not happening anytime soon? :)

Yah, that's probably a good idea. Maybe I'll try the middleware-style approach again and then if I can't make that work in a reasonable way fall back to parsing the entire stanza up front.

The reason why I haven't focused too much on it is that it simply isn't the bottleneck. In Prosody, I can do 100k stanzas/sec on this old macbook. And profiling does not put memory allocation as being item #1.

The string interning Lua does probably helps a fair bit

(standard microbenchmark disclaimers apply)

Zash: No, those don't save us. While it has a few tricks, expat is doing many allocations.

In our case, the networking stack is bottleneck #1 (including the kernel, luasocket and net.server), the whole parsing stage (expat + DOM construction) is #2, serialization is #3. All this for a very simple full JID to full JID IQ. Doesn't change much for other stanza types, except stanzas that cause broadcasts and IO are complex.

Right now for me parsing JIDs is the most expensive part of the stack, but I've gotten that down to be fairly quick (especially if the JIDs mostly end up being ascii)

Turn on any IO activity (MAM, logging, etc), and boom, it suddenly overshadows everything else combined

Heh, like debug logging

Debug logging is a fraction of my I/O, compared to gazillions of chatstates in MAM.

In a former life though parsing the token stream into some form of usable object was very expensive though, and I'd like to shoot for that sort of scale. I can't do it right now because of the XML library, but it would be nice to be able to do it if I were to swap that out in the future.

I'm pretty sure most of my overhead while dev'ing is the drawing of syntax-highlighted stanzas in X

Overhead for clients and servers is completely different.

The most expensive thing Swift does is rendering a roster.

Well, s/rendering/constructing and rendering/

If I were start today, I'd go with some dynamic language that has primitives to convert between objects and their stream representation, and ignore performance.

Kev: That's very true. And many clients do synchronous UI updates. Gajim and others do very badly when a bunch of stanzas show up..

...and then end up with a situation where remote entities can override my objjects' metadata pointers.

Indeed; luckily I'm not doing any of that, but I do get frustrated when Gajim completely locks up for a minute or so during the initial presence flood on a big roster

Swift does synchronous UI updates, but each one is injected into the UI's event loop individually, so unless the UI framework (Qt) is being daft, you'd need one very heavy stanza to freeze the UI.

Pushed up my current experiment which didn't end up working out, in case anyone's interested: https://godoc.org/mellium.im/xmpp/mux

The idea being that you match on the start token and could nest the multiplexers, but in practice you always need the top level stanza's information in whatever handler ends up replying.

hmm mux was the name I was rooting for as the mix replacement people actually implement to replace muc with

oh yah, I would probably be sad if I called this package that and then ended up wanting to implement that later. Maybe "router" would have been a better name for this one. Oh well, it will end up being deleted anyways.