XSF Discussion - 2017-09-15

Link Mauve: to late 😉

who can set me up with access to the XMPP Twitter account?

moparisthebest: I wondered because when I looked up nick hash coloring recently I saw that SHA1 was used to calculate the hash.

tux: sha was my first choice indeed, but it doesn't really matter

are there other implementations which use crc?

tux: I haven't checked

jonasw, I wondered about CRC32. It feels like it's more prone to accidental collisions. Adler32 is less so, and faster. And MD5 even less so and fast enough these days.

dwd: Adler doesn't work at all in those input sizes

If we could come up with a hint that one of the algorithms is most common for nick hashing I'd vote for that most common algorithm.

dwd, it doesn’t matter a lot (except that Adler32 needs much more input to be effective) since we fold the output to 16 bits anyways

tux, interesting idea; however, my intention was to go with something which is easily available on all platforms and implementations; the zlib crc32 should be such a thing

with MD5, we had the concern that it might vanish from crypto libraries at some point

For C/C++ that's true. I'm not so sure about other languages.

IMHO it would be sensible to have a scheme that is viable to as many (modern) programming languages as possible. We want people to use this.

most languages have some kind of zlib support, because the zlib is everywhree

I agree

I can check how well Adler16 performs. we might be able to seed it properly so that it randomizes, but I’m not too confident in that. the adler concept is a bit weak on the mixing side

jonasw, I think CRC32 predates zlib, actually.

dwd, sure

The fine thing about CRC ist that it is easy to implement

is it? I never managed to :)

Huh. CRC is 1961, CRC32 is 1975.

dwd, I’m referring to zlib crc32 because of the specific zlib polynomial used

At least I did some working implementations. ;)

tux, in my defense, the platform I tried it on was VHDL ... soo ... ;-) but given that there exist libre implementation of the zlib CRC32, it would be easy to port to any language, if there are really any which don’t have access to a CRC32 implementation with that polynomial

jonasw: yes, it is. Maybe it makes sense to link to a normative description for CRC32. I'll have a look during the weekend.

do you guys know XMPP clients that are doing Jingle with WebRTC ?

yes, I hadn’t found one

edhelas, jitsi maybe?

tux, if you could find a nice normative reference for the zlib CRC32 or CRC32 in general and the zlib polynomial, that would be of great help

jonasw, No, Jitsi-the-desktop-client doesn't interop with JitsiMeet.

dwd, aw, pity

put them on a stake and burn them with flames

:)

edhelas, The only ones I know of are bespoke Web ones.

(just kidding)

dwd, that's what I thought

The benefit of MD5, is that every XMPP implementation already knows how to do that.

jonasw, "Classic" Jingle VOIP doesn't interop with WebRTC Jingle VOIP.

so we don't have any client, except Movim, that implement Jingle using WebRTC

intosi, isn’t it a MUST NOT by now? :)

intosi, True. Also there are loads of implementations everywhere.

jonasw: sure, but legacy.

intosi, will legacy implement new things?

jonasw, In security cases.

Fine, then propose SHA-1, which is definitely required, for things like CAPS.

and I’m sure those implementations depend on lib*ssl for those; will lib$whateverssl support MD5 in the future?

(SHA1 will probably work better, indeed)

(support-wise)

jonasw, There's a number of issues with MD5, but the worst one now is speed. MD5 is so fast that brute-forcing is practical. At the same time, and increasing number of people are using it in this kind of thing because of that.

dwd, fun fact: sha1 is faster than md5

fun fact: BLAKE2 is faster than SHA1 :)

dwd, https://sotecware.net/images/dont-puush-me/9n701UwrIuU2NFuTexy7rFpu3Auf7zbQlVsEyFTVqAM.png

If speed of nickname colour hashing is going to be your issue, I wonder how your client is to keep up in such a room with all the messages ;)

intosi, frankly, *I* do not care much about it; my client will cache the colors anyways.

but I heard that it’s not that trivial for other implementations.

jonasw, I thought it was only faster where the silicon supports it? But in any case, SHA-1 sounds good then.

dwd, the difference seems to small for silicon support to be the reason in this graph, I think. so yeah, seems to be faster

(not that it’d matter on our input sizes)

I like hashing methods that are already in use in code ;)

Yeah

intosi, To be honest, I'd go for SHA-1 over CRC32-with-some-polynomial just for the ease of specification. ✏

I tried to calculate my nick colour yesterday evening, and wasn't sure whether I was using the correct polynomial.

I’m all in for that

As in some environments, folks just mention it's a crc32, and never bother telling you which one it actually is ;)

the question is: shall I update the protoxep or update it when it goes experimental?

intosi, On this client, it is, rather fittingly, orange.

👍for sha1

Orange is good.

intosi, You say that, but it's not working out well for the US.

dwd: yeah, but they're holding it wrong.

lol

jonasw, Just wait.

dwd, great, that fits my schedule better

Yeah, we should provide a sample set for software testing. With hashing methods it's always kind of a guess whether the results are correct.

tux, there are test vectors

Ah, okay.

jonasw: hm.. You mean test vectors for CRC32 or for the Nick coloring?

the latter

Ack

edhelas, Jitsi Meet is a Jingle client, just like Movim.

'p

Link Mauve - do you think Jitsi Meet will do interop Jingle? I know that they recently added p2p support, but otherwise, everything typically flows through the videobridge. Jingle is involved there, but is it functional outside of their own solution?

It'd be great if they do though

does the visioconference part of Jitsi do even pass by Jingle ?

looks like their solution is not a full XMPP based solution

edhelas: The Videobridge is XEP-0340.

Holger, okay

oh, there's definately a lot of XMPP in there. I'm not sure if it's interoperable, though, with non-Jitsi clients

Guus, I believe that Talky uses it well enough.

(sorry, wrong window and tab completion fubar) ✏

(he's talking about you in another window, Dave)

talky is now based on a bunch of deferred standards :-)

I might have to end it now I know this

Haha

mattj: there is also a proprietary namespace used. nothing critical at least :-p

fippo: isn't everything based on deferred standards? :'(

even the 2017 compliance suite specifies deferred standards, or at least ones that have been in last call for months and should be deferred

the internet runs on drafts

fippo: 👍

regarding the coloring thing, I’m still uncertain whether using the roster name or the JID is preferable. using the roster name has the advantage that multiple JIDs to the same contact can have the same color. using the JID has the advantage that the color is stable even if I change the name.

opinions?

using the JID also has the advantage that the three different Martins in my roster could have different colours

and I wonder how relevant the "I have the same person with multiple different JIDs in my roster" use-case really is

I'm a fan of the JID. Although it would be a bit weird because toys have to do the bare for 1:1s and the resourcepart only for mucs probably.

"toys"?

Weren't public semi-anon MUCs an edge case only used by nerds who nobody cares about (ie us)?

for MUC the situation is indeed so that the nickname is used if the muc is anonymous, otherwise roster name or JID, depending on what we decide on

JID sounds really nice to me.

I also tend to JID by now, even though I first thought that roster name would be nice

No opinion from me

*sigh* phones

I think that was supposed to be "you would have to use…"

Also what Zash said though; probably best to just ignore MUC to keep things simple. The XEP already *looks* too long for such a simple client feature (I read through it, it seems fine, but initial impressions are everything and I'm already concerned that it looks complicated enough that clients won't bother trying to implement it)

SamWhited, I thnik on the other hand that many XEPs are way too short because they lack detail.

the part on MUC is literally three short paragraphs, I wouldn’t omit that

SamWhited: MUC nick coloring was the actual reason for inventing that XEP

we can move the test vectors to some appendix or so and not put each case in its own section, that’ll shorten the TOC a bit

jonasw: It's not a big deal, it looks like a really good XEP overall, I just worry about these things.

thanks, and ok

(also the thing isn’t that simple anymore once you consider color vision deficiencies and such)

Maybe I should read the XEP as well, instead of just relying on what I think it contains.

Ge0rG, haha :)

:D

Ge0rG: Blasphemy!

SouL: something I had to explain to a customer today: as a Ph.D., you learn to make confident statements about something you have no idea of.

Ge0rG: You better only look at the examples!

-xkcd 793

Physicists https://xkcd.com/793/

actually, I just rely on jonasw writing down what we discussed at length multiple times ;)

I pretty much do only look at the examples the first time I "read" an XEP. I figure that's what client devs are going to do, so if they're confusing or lead to the wrong behavior it's probably something that should be fixed

what hurt most while writing the XEP was trying to stick to en_US ;-)

Yeah, blaming client devs for (even major) failures is very disappointing and surprisingly hard work. BTDT.

I'm not actually sure how to read that; if it sounded like I was blaming them, I'm not. I think that it's fine for them to only glance at the examples and it's our (XEP authors') fault if we don't communicate clearly with our examples.

Ge0rG, super good one :)

SamWhited: you are absolutely right, my remark was rather tangentially related than a direct answer to yours

*nods* I think we're on the same page

It just made me remember the Carbons attribution issue from earlier this year, and I think the Carbons spec is rather clear.

At least in that regard.

ah yah, that one was bad

And so were the reactions from many developers. ✏

Were they? The one or two issues I submitted (Mcabber and Jitsi, I think) were pretty good

SamWhited: some developers just didn't care

figures

*sigh* oh well.

Ge0rG, did you send them carbons claiming to be from themselves to confuse them? :)

jonasw: I sent some, but not to the devs

Or at least not claiming their identities

why not? :( ;-)

It didn't appear to me, back then

Instead I claimed to be an unpopular political character.

What would be the least inelegant way to implement a self-ping feature on MUC JIDs? It needs to be discoverable via caps info, and it needs to be something that doesn't have meaning yet. Maybe a certain IQ to the bare JID?

Discoverable by caps? That's hard.

Ge0rG, make a feature var which defines a ping to the bare MUC jid to only succeed iff the requesting resource is joined and return <item-not-found/> or some other error if the user is not joined ✏

jonasw: that would probably make sense.

currently how that ping behaves is undefined-ish I think

we need to feature-check it anyway.

yupp

jonasw: currently I'm pinging my own participant JID, which is a double-RTT va banque game.

I know

jonasw: I think that ping IQ semantics are defined already, and changing them is actually incompatible. Maybe a dedicated IQ makes more sense?

Ge0rG, is the ping to a bare MUC jid really defined?

daniel, you have some insight in the OMEMO topic, right? can you confirm that this is the result of the recent discussion which should be merged? https://github.com/xsf/xeps/pull/482

Ge0rG, What are you doing to do with this ping?

jonasw: Oh, right. IQs must go to a full JID

Ge0rG, Erm, no.

dwd: I need to figure out if I'm still joined to a MUC

Ge0rG, OK. And if you're joined, you'll do nothing, and if not, you'll rejoin?

dwd: exactly

Ge0rG, So - and I admit this has inefficiencies - as a first step, could you not just send presence?

Ge0rG, the ping to the MUC component domain makes sense, the ping to the occupant JID does too, the ping to the bare MUC jid doesn’t really

dwd: no.

Ge0rG, Why not?

dwd: sending presence is a "groupchat 1.0 legacy" join ✏

jonasw: not practically, but technically.

Ge0rG, sure

Ge0rG, Well, yes, you'd get unwanted history potentially. But if that's *quite* close, is it somewhere to build from?

and if the behaviour is practically undefined, you can re-define it to be something useful with a feature var

dwd: the MUC implementation might mark me as a legacy client.

Ge0rG, and sending presence with <x/>?

jonasw: would do a full rejoin every time.

ugh

even for a joined client?

jonasw: you remember my last PR? ;)

yeah

it was exactly about that.

do implementations behave that way?

I have no idea. I think prosody trunk might.

jonasw, Rejoin handling? M-Link, Openfire, possibly ejabberd.

Ge0rG, When you say "might mark me as a legacy client", what do you think this would do? No servers I'm aware of do this.

dwd: I have no idea. I consider "legacy groupchat" to be something to burn with fire, or at least to avoid.

dwd: if I were God of XSF, I'd completely remove legacy groupchat from 45.

I sorta wanna remove it from Prosody. IIRC it'd simplify somewhat.

dwd: I have no idea if anybody is still using it, or if there are dragons inside the implementations.

MUC got enough hairy corner cases without groupchat legacy.

What is "legacy groupchat"? I don't see any mention of that grepping for "legacy" in 0045

Though I must admit, "Groupchat Legacy" would make for an awesome title for an XSF-themed movie. Or computer game. Or rock band.

SamWhited, search for Groupchat 1.0 Protocol

SamWhited: https://xmpp.org/extensions/xep-0045.html#enter-gc

ahh, yes, thanks. Kill it with fire.

Ge0rG, The only difference is the join, though. Once you're joined, its exactly the same.

SamWhited: I volunteer to make the diff to 45, provided that council will signal pre-approval.

Ge0rG, So by periodically reflashing presence, you're just risking getting the wrong history.

dwd: Like gmail did?

dwd: will my presence get forwarded to all participants?

dwd: I have a MUC with histsize=500.

Zash, Ah, gmail re-sent a '45 join periodically as well. That was mega-annoying.

Rebroadcasted all directed presence too iirc

Ge0rG, Yes, and also erk. But I think it's an interesting place to start.

I think a ping is a cleaner solution. nothing wrong if it’s with a feature var, is there?

In such a way that if they changed nickname, they'd periodically switch back and forth

Ge0rG, If your server both maintains, and always sends by default, 500 lines of history there's something badly wrong somewhere.

Zash, Indeed. I'm not suggesting that.

dwd: I totally could live with a <presence><limited history><I'mtherealready></p>

Ge0rG, In principle, a server could/should not rebroadcast presence where its identical anyway. But that's a small point.

dwd: I think that having 500 lines of history in the default buffer beats MUC-MAM.

Ge0rG: I can't speak for anyone else, but I'd be all for that.

dwd: it's a small O(N²) point.

Others are probably more conservative since it is in draft, after all.

Eternal draft.

<presence><{muc}x><{mam}history plz/><//

Ge0rG: Is there some reason to move it on past draft?

SamWhited, The fallback is it's a rejoin/rebroadcast.

dwd: there is another problem with the presence approach: if you lost sync, you'll get the history (mkay), and you'll get a full participant list. But you need to know in advance that you are receiving a full list so you can clear your previous list.

dwd: otherwise you'll end up with stale participants in your cache

and race conditions will prevent any sensible workaround

right

Besides of all that, some client (library) implementations don't expect to receive a join from a MUC they are already joined into. So they might lose sync anyway.

Ge0rG, Well a library written to use a periodic presence as a ping-or-rejoin will work fine, surely?

dwd: yes, unless you hacked the ping-or-rejoin code on top of an old library. ;)

Maybe it would be cleaner to respond to a non-x-ed <presence> with an error, so the client can rejoin?

But still, we have the O(N²) problem that every client, sending a periodic presence, will flood everybody else.

Can anybody gather stats on "groupchat 1.0" usage?

https://matthewwild.co.uk/uploads/gc_pinger.html

(not an answer to your question about usage stats)

Ge0rG, "If I write my code badly it might be written badly" is not a tremendous argument.

MattJ, You're suggesting that this be done server-side instead?

I'm merely dropping a historical anecdote :)

dwd: it's not a tremendous argument indeed. Please focus on my two material arguments instead.

Ge0rG, Excellent. So if the client is dropped from the MUC without its knowledge, and simply sends an updated presence, then you'll be inadvertantly getting the same problem but this time it's unavoidable.

dwd: right. But this is another argument.

Server-side doesn't tell the client that they are disconnected, does it even help? It helps them not be de-synced when they rejoin, but only if the server happened to ping them in that time period

Ge0rG, The only solution would be to, before updating presence, send your IQ-based ping beforehand. But then there's a minute race condition, which you'll no doubt be disproportionally upset about.

dwd: I don't think it is wrong to focus on corner cases when aiming to design robust protocols.

dwd: if we design a 90% protocol, we'll end up in the place we are in today.

+1, corner cases matter

where my desktop client at home disconnects from a MUC because I'm joined to the same MUC with the same nickname from my mobile device when on the go.

If we design a 100% protocol, we'll be done in the year 3000?

Zash, then we can release 0.10

Ge0rG, Excellent. So I've given you a corner case which your design fails to address, and my (sketched) design does.

Zash: fixing MUC isn't _that_ complicated. If we can get rid of legacy.

Ge0rG, Legacy meaning gc?

dwd, and how to solve the issue with inconsistent occupant state?

I'm inclined to agree with finally dropping GC 1.0 support

when I’m re-sending presence and in fact I’m doing a join, I don’t get presence notifications for occupants which left in the meantime

dwd: I'm not saying your design is bad, or that it's wrong to follow that approach. I'm just pointing out that your sketch won't work.

I think both designs don’t work

dwd: in the current context, legacy means GC. But I'd be glad to kick out more legacy when appropriate.

and ideally there would be an IQ based join which either fails or succeeds and only after the IQ returns you get the presence updates, so that it’s fully synchronous

jonasw, Sure, but that's MIX.

jonasw: I'm okay with a presence based join that succeeds as soon as you get a presence reflection.

Which is MUC.

dwd: I could imagine a periodic presence with yet another additional element, where the server would respond with a presence-unavailable, followed by a full rejoin.

dwd: and all of this would depend on a feature flag.

dwd: and the server wouldn't forward such a presence to other participants, unless it contains a delta.

dwd: the presence-unavailable (or any other response marker) is needed to make the client flush the cached data.

Ge0rG, "succeeds as soon as you get a presence reflection"? I don’t understand that part

jonasw: a MUC always reflects presence to you. So once you get that reflected presence from the MUC, you know you are inside.

reflects your own presence, that is.

sure, but ... how do you know the occupants list when you didn’t know that you re-joined?

jonasw: "the presence-unavailable (or any other response marker) is needed to make the client flush the cached data."

ahh

(Or: The occupant data could have a delay tag in)

dwd, does that help you to identify which occupants have left in the mean time?

1. MUC advertises "smart-resync" 2. you join, normally. 3. you send a <presence><smart-resync> 4a. you get a presence reflection 4b. you get a presence-unavailable followed by a full resync

No, but you know when the end of the occupant list is, so that should be OK.

dwd: you need to know the beginning of the occupant list.

dwd: so essentially you must strip <delay> from inbound client presences

Ge0rG, What?

and if the client receives a presence with a delay, it knows that it's at the beginning of a rejoin.

Was there a status code for 'this means you just joined'

Ge0rG, I suppose you could do it that way. Or spot your own afterward and clear any untainted. Or ...

Zash: yes, it will be in your own reflected presence at the end of the occupant list

dwd: RACE CONDITIONS!!1!!

Sorry, I just couldn't resist.

Ge0rG, I don't think there are any there.

Ge0rG, Unless you're getting stanzas out of order.

dwd: any participant presence you receive after sending your smart-resync-presence might be either old presence from your previous MUC session or the beginning of the occupant list. No way to know.

So what you could do is to put all the presences into a queue until you get your own reflection. Or the subject.

I think it's your own with the appropriate status code

Ge0rG, You'd get your own afterward, certainly. And any presence you get after a smart-resync is either un-delayed, in which case it's the first stanza(s) you'll get and it potentially stale but who cares because you'll notice this later - or else is delay marked, and therefore part of a resync.

dwd: unless it is a delayed presence because the sending client added a delay element.

Ge0rG, The only pattern, in other words is: *non-delay [ *delay ] your-reflection

Ge0rG, Yes, but there's a thing in delays to tell you who is delaying.

dwd: and I need to put all the non-delayed presences into a queue and unwind them once I realize it's a resync.

dwd: that's not how you design a robust protocol.

dwd: and really, what's wrong with injecting a presence-unavailable?

Ge0rG, No you don't.

So what isn't solved by <presence type="error"> if the client isn't in the room and they send presence with no <x>?

Ge0rG, The moment you see a presence delayed by the MUC, you're rejoining. You weren't before, you now are.

dwd: okay, you try to make it work in a backward compatible way. I appreciate that. But it's not easy.

MattJ: groupchat 1.0

and apart from that?

MattJ, Breaks compatibility with gc, of course. Whether that's now safe to do I'm not sure, though it feels dangerous.

What clients do gc?

I'm willing to do it

What *clients* do it, not counting advanced users manually typing XML in netcat (like I do sometimes)?

I don't mind if 20 year-old clients break, and probably some bots. Any actively maintained clients likely do the right thing already or will be fixed promptly

MattJ: you could create stats on GC1 clients.

MattJ: presence-error would also be a nice way to make GC1 client devs care.

Prosody trunk MUC logs such conditions already

Debug level, but still

most of it will probably be resync problems.

Does/did gc have any discovery? I suspect not.

Ge0rG: It logs joins without <x> and presence with <x>

Okay, I'm late for my weekend already. Thanks very much for the productive discussion. Maybe you can bring up killing GC in the next council meeting?

I think that's the best solution, and I'm willing to move forwards on it as a server dev

I think that would be a good first step, with the second step being an explicit smart-resync presence feature

fg

I'll add it to the agenda when I get back to my desk if someone else hasn't already (leaving this here as a note to myself)

kill -STOP %1

Now he'll never get home

MattJ, "SIGSTOP cannot be caught or ignored."

SamWhited, did it for you

Thanks

who has power over https://hub.docker.com/r/xmppxsf/xeps/builds/ ?

I could use someone hitting a retry button.

that build failed due to some remote network issue

jonasw, Randomly update the README?

good point, I wanted to merge my readme stuff anyways

woah jonasw warning: failed to load external entity "http://xsltsl.sourceforge.net/modules/node.xsl" the xep build depends on sourceforge being up? that sounds like a bad bet

can it be fixed to not depend on any remote entities?

Hey, did Thiago become active again?

nice change @ jingle relay nodes.

mattj: remember that xep-0215 is much better though!

edhelas, DNS resolution failed on movim.eu

which DNS ?

movim@conference.movim.eu/pep.: cancel: Server-to-server connection failed: DNS resolution failed

I can dig it from here actually :/

And from my server

IPv6 maybe ?

works as well