XSF Discussion - 2018-02-12

daniel, Matrix already does that. http://mirror.onet.pl/pub/mirrors/video.fosdem.org/2018/H.1309%20(Van%20Rijn)/matrix_webvr.webm

looks cool

daniel: Psi gets crazy when it receives muc self-presence, it renders "empty" participant in muc roster

O_o

yeah

What server should I install to have such a self-presence? ejabberd?

rion, no, it's on local machine

in any case last two day I rewrote muc roster. so it won't happen anymore. not commited yet

good

the problem is that now I'm not sure if other clients don't break though

rion, Psi requests MUC vCard on every join?

yep

I think with self-presence and caps I'll change this too

and I have to improve avatars caching for mucs too as well as their requesting algo. otherwise it's a lot of traffic on join.

yeah...

poezio has the same problem wrt avatars 😉

gajim seems doing fine, doesn't render anything strange

By the way, regarding the email I sent to JUser, is there a better mailing list for that, please?

I still haven’t subscribed to juser :/

That's why I would like to send it somewhere else too, seems not many people are subscribed there

Seve, I think operators would be more fitting, maybe

possibly with a cross-post to members@

I’m also not sure if a separate MUC for this is a good idea

Posted something to some KDE list?

frankly, I’m not sure XMPP can fulfil the requirements at this instant in time

> Stickers are available

we doomed

Find out which are requirements and which are wishlist

Zash, they don’t even know that yet

> Requirements are not prioritized (incl. must-have vs. nice-to-have), that comes at a later step.

jonasw: All are wishlist. There's online one requirement, that everyone you care about is already using it. :)

jonasw, the point of the MUC is to discuss this topic with people interested in helping out. Almost everybody here would find this discussion annoying here.

Seve, I’m running out of screen space for more MUCs.

Ge0rG, jonasw can you review https://git.zapb.de/xeps.git/commit/?h=fix_xep_0401&id=52725e993987f205dc253cc5a4e6937fe3955d81 and merge please?

marc, EBUSY right now

marc, if you refuse to make a github PR (which would really be the most useful thing for me), please send an email with the first line of the commit message as subject to editor@xmpp.org

jonasw, I can do a GitHub PR if you like

marc, that would be much appreciated, thanks

jonasw, you're welcome

gotta run now, see you

bye

done

zinid, regarding the server capabilities invalidation, how about a message?

server changes caps -> server sends <message/> to local clients. for server-to-server links, it could simply close the link so that the peer re-opens it, thus getting new stream features

jonasw, closing and opening thousands of s2s will create avalanche effect

right

we see this on jabber.ru for examlpe

so I just think about nonza

stream-level element

zinid, thanks for your input, I’ll post a few suggestions to the mailing list and I hope you’ll comment on them :)

sure 😉

Is there a good resource on the problems with message ids?

zinid, replied, happy to hear from you :) ✏

SaltyBones, I’m not sure

I'm looking at XEP-0359 but the problems mentioned at the summit are not in there.

maybe the summit notes as first starting point

1.2 Stable IDs Discussion on entropy ensues. dwd: <stanza-ids> don't work for unique addressing because we don't trust clients to do them properly. Kev: lots of possible attacks with spoofable stanza IDs.

Not verbose enough.

SaltyBones, I think the gist of that is: we need to generate stanza IDs on the server because weak/constructed stanza IDs are a problem. but the client needs to know the stanza ID for archive operations and possibly other things

jonasw, well, dunno, for me message looks too complex

can't we just send <c/> as a nonza?

zinid, I think RFC 6120 would slap you in the face for that ;-)

really, it needs negotiation

it’s a shame that we have no way for a client to send stream features :(

yeah, because some implementation may close a stream if they receive unrecognized nonzas

yupp

I know at least one.

however, I tried to implement such behaviour (closing a stream) and got lots of problems and left the idea

never had issues so far; did you encounter things which sent unsolicited nonzas?

alas, I've already forgotten what that was exactly 😕

zinid, the best we could do is probably say "okay, if the client sent a presence update with caps, we can send a nonza with a caps update"

and similarly for servers ("offered caps in stream feature -> send nonza")

but I don’t know enough about s2s to be sure that this works

this would definitely need a namespace bump though

well, we're talking about new caps xep?

we can do it there

yeah, it requires a namespace bump there too ;-)

can't we just negotiate this feature?

not sure how

negotiation will add a round-trip

which people will react very adversely to

ah

but please comment on-list, hoping to get more input on that

I don't know what to say

what you said here, essentially?

There are stanza-id and origin-id in XEP-0359. Somebody at the summit also mentioned "message-id" is that defined somewhere?

They were probably referring to the id attribute, I guess?

MattJ, this https://tools.ietf.org/html/rfc6120#section-8.1.3 ?

Yes

thx

The reason XEP-0359 exists is because the id attribute is controlled (and can only be trusted by) the original sender of the stanza

It's really just designed for tracking errors, though a couple of XEPs have re-used it for other purposes

The id-attribute you mean?

yes

So, I guess, stanza-id is used by MAM and origin-id is used to detect MUC reflections (that's what 0359 says anyway). And id-attribute is used for errors but the error will have the same id-attribute iiuc, right?

yes, pretty much

except that origin-id won’t work with all MUCs

Why?

because not all MUCs may be able to reflect that ID for whatever implementation specific reason

(just like not all mucs can reflect the message @id)

There is only one implementation I'm aware of that doesn't (didn't?) reflect @id, and I'm not even sure of the current status of that

Okay, so why does the standard not just mandate that it be reflected?

Simple oversight I suspect

SaltyBones, Non-MUC things pretending to be MUC, in part.

SaltyBones, Also by the time people had noticed this was a problem, enough implementations didn't.

So, could this be fixed by changing the standard and requiring that origin-ids be reflected?

Dave Cridland, what do you mean by "non-muc things pretending to be muc"?

SaltyBones, changing the standard doesn’t fix the implementations magically

Dave Cridland, enough = 1? Also in the case of bridging to non-MUC, isn't it as simple as 1) if the non-MUC supports acks, wait for the ack and reflect the id or 2) reflect the id immediately?

SaltyBones, IRC transports are "non-muc things pretending to be MUC" for example

jonasw, not magically but ...

MattJ, More than one, I think. One classic MUC implementation, but quite a few transports.

FWIW, I've gone back and forth over whether MUC ought to reflect ids. It's a special case of maintaining the id on bradcast, which is itself both useful in some cases and bad in others.

Why is this reflection necessary?

SaltyBones, Reflection isn't - we could add in a subelement which indicated the original id.

SaltyBones, Broadcast ids are harder to work around - there's a few protocols which use id as reference.

This all said I *really* dislike having a zillion ids as subelements of stanzas when we already have an id attribute.

Well, what is the subelement necessary for? Is this just as an ACK to the client?

What do you mean by broadcast-id?

SaltyBones, if you're asking why MUC reflects messages to the sender in the first place (some systems, e.g. IRC don't) - it has a few benefits

Such as ensuring everyone in the room sees the same messages in the same order

it also is natural as soon as multiple clients are in the play, somewhat like message carbons

SaltyBones, So the id attribute in the message stanza for a groupchat message sent to the MUC can be reused in the reflected message, and/or the broadcast messages to other participants in the group.

In IRC if two people send a message at the "same time", the messages will be shown in a different order on both their clients

and yes, multiple clients from one user in the room

it allows MUCs to do fancy things to messages and still have everyone see the same thing

(As a side-note, Matrix achieves a similar goal by a complex hash chain)

SaltyBones, and also servers modifying messages (e.g. in the Prosody MUC we automatically convert long multi-line messages to a pastebin link). The reflection allows the sending client to see what everyone else sees.

(in contrast to IRC, where you don’t see that your message was truncated)

Okay, pretty convincing...

Dave Cridland, and as many people at FOSDEM noted, much RAM

MattJ, Yes but BLOCKCHAIN.

#bingo #triggered

🔲‍⛓️

I hope there's a ZWJ there.

*Yes, there is.*

Okay, so we want reflected messages. Sounds fair. Let's suppose for a second that we can just make them mandatory in the standard and people would add them.

they *are* mandatory

the issue is with message IDs

clients need to be able to recognize the reflection

You mean id-attribute?

or any ID really

id attribute would be easiest

We could indicate in disco if ids were stable.

wasn’t that proposal rejected-ish?

Okay, 1. is the origin-id used for anything else and 2. is the whole point of origin-id not to detect reflection?

speaking of MUC, I think the vote on https://github.com/xsf/xeps/pull/559 expired, didn’t it?

I've attemtped to fix MUC reflected IDs some years back, not even in the normative language but merely by fixing the examples

And got significant flack for it.

I also attempted to make origin-id == message-id, but it was refused by the XEP author.

Now I'll just sit and wait, and ocassionaly proclaim: *I told you so!*

So, is the point of origin-id to be used for reflection or is there something else?

SaltyBones, that is one use case

Ge0rG, great, if you could also occasionally chime in with reasons for things and explanations that would be awesome.

Flow, like?

SaltyBones: it's mainly for reflection, except that MUCs are not guaranteed to keep non-body elements in the reflection

other use cases include finding your own messages in a archive

SaltyBones: https://wiki.xmpp.org/web/XEP-Remarks/XEP-0045:_Multi-User_Chat#Matching_Your_Reflected_Message

Flow, MAM uses stanza-id, right?

SaltyBones, it does

SaltyBones: yes, but you don't know the stanza-id for the messages you sent

Ge0rG, would you say this is a requirement by nature or just a problem with implementations?

i agree that the xep should specify that the sender should set origin-id=message-id

SaltyBones: what?

daniel: MUST set.

Ge0rG, the fact that MUCs don't reflect non-body attributes.

yes

because anything different is just a path into insanity

(i didn't mean SHOULD)

Ge0rG, also, why do I need the stanza-id for a message I sent?

SaltyBones: have a look at biboumi, a modern IRC transport. It doesn't reflect your message ID, it doesn't reflect any non-body elements and it mangles multi-line messages

Welcome to message tracking hell.

i think thinks will already break if some client doesn't do that and then expects a delivery reciept or something

SaltyBones: because when you ask for a MAM sync, you'll get yout sent messages copied to you as well

when sending messages to Conversations i mean

Yes, message correction, delivery receipts and anything else that references IDs is a mess already.

Ge0rG, and why is that a problem? They should have their origin-id and be recognizable, right?

SaltyBones: you can't rely on origin-id being there, and you can't rely on it matching the message-id.

daniel, what breaks when a client does what?

SaltyBones: but yes, you can match MAM archives based on origin-id

if an origin id is set Conversations will use that as a reference in the receipt

so if your client doesn't do id=origin-id and expects the receipt for the id it won't work

same with everything else that references ids

So, there seems to be consensus at least in this MUC right now that attribute-id should be the same as origin-id?

i'd argue there is no good reason not do make this a MUST

i mean most client would naturally do this anyway

Ge0rG, who was the xep-author who was against this and do you remember why?

SaltyBones, Flow

because why would you generate a second id if you can reuse the existing one

Who is generating origin-id? A client?

SaltyBones: https://mail.jabber.org/pipermail/standards/2017-September/033415.html

zinid, the original sender

zinid, yes

i.e. most of the times a client

note that muc rewrites a irrelevant to the scenerio described above

jonasw, #559 expired, yes.

This is to track its own messages?

zinid, yes

or references to that

jonasw, But with no veto and the rest are +1, so apply it.

jonasw, what is this ID-rewriting MUC shit? :)

some mucs are known to change the 'attribute' id

SaltyBones: have a look at examples 44 and 45 in https://xmpp.org/extensions/xep-0045.html#message

so tracking your own message are parsing references don't work

daniel: why, you can rely on origin-id in this case

If we have a MUC that rewrites message IDs, can we mandate that it also MUST rewrite references in all XEP payloads that reference message IDs, please?

zinid: https://wiki.xmpp.org/web/XEP-Remarks/XEP-0045:_Multi-User_Chat#Matching_Your_Reflected_Message

zinid, in the case of muc? because most mucs won't remove child elements from the stanza

I'm lost

daniel: but that's exactly what's needed

now i'm lost :-)

So you can fetch origin-id and check if this is your id

Why do you need to rely on message-id if you inject origin-id

i just said the sender should set id=origin-id

zinid, I think there was a sub-discussion about forcing message-id = origin-id

and/or deal with delivery receipts that reference the origin-id instead of the id

which is made easier if you id=origin-id

Ok, I don't get it 🤔

doesn't matter. that client to client stuff anyway :-)

From what I understand we just need to ditch IRC transports

😃

That's really their problem

oh yeah. or make irc transports track the messages themselves and re-add origin-id and stuff on reflection

it's probably better if the irc transport does the right thing(tm) than letting each and every client figure it out

daniel: agreed, nobody said writing transport is simple

irc transport should also reassemble the message if they previously split it and stuff like that

because they know best if they did

Hm.

Okay, are there any actual problems with the current state of three IDs except that we don't like having so many?

yes

read what i said before

Uh, assuming a well-behaved server?

yes

my argumentation has nothing to do with servers

daniel, IRC doesn’t even reflect

so the transport generates the reflections on its own

Okay, can you please point out what you are referring to then?

biboumi however decided to reflect the split version of the message, and there’s some argument to doing that

assume i sent: <message id="1"><body>hi</body><request/><origin-id="2"></message> and then i will receive from conversations: <message><receipt id="2"></message>

who cares about IRC, really, are we designing a protocol for old nerds?

but we are talking about two different things. the how should irc transports behave has noting to do with the problem i'm describing

this applies to 1:1 chats as well

daniel, yes, I don’t argue that

I’m just replying to your message from "13:49:15Z" because I was away from keyboard.

the irc transport with the name i can not pronounce or remember does a few things that i don't like :-)

daniel: if we don't care about IRC then we probably don't need origin-id and thus there is no the problem you just described

there are non transport mucs which also rewrite the id

daniel: https://lab.louiz.org/louiz/biboumi/issues/3283

zinid, if it weren't only for transports i wouldn't care

Ge0rG, is that an issue to change the name to something i can remember?

or pronounce?

daniel: if they are abandoned, then I personally give zero fucks

daniel: no, it's about maintaining IDs on reflection

zinid: everything in XMPP is abandoned. Now stop giving fucks.

i should probably write my own irc transport

biboumi is not good ?

daniel: you should just patch biboumis two or three warts.

i haven't used irc though ever since counterstrike 1.6 came out

Ge0rG: not everything, so I still have a few fucks to give

there are a number of things i don't like that seem to be design decisions rather than bugs

so i'm not sure if they even want me to change them

daniel, so regarding your example, what you are saying is that the read receipt might reference the message-id or the origin-id and it is not properly specified which?

Ok, whatever, so what do you guys think about muc subscriptions?

Or are you getting at the fact that origin-id requires "by=" and is therefore sometimes not applicable?

> daniel, so regarding your example, what you are saying is that the read receipt might reference the message-id or the origin-id and it is not properly specified which? This

not just read recepits but everything that references something

but yes

daniel: isn't there a place in the xep which says you should copy the message-id?

zinid, no.

Really?

that's what Ge0rG and I want to add to the XEP

that's pretty much what the entire discussion is about :-)

So add it 😀

i can't without permission of the author

No shit?

So, there is some problem that I am still not aware of....

Flow, why is the by attribute in the origin-id?

Who's the author?

The one that causes the privacy problems...

Not entirely true, BTW, that it's impossible to do things without the author. But it's the path of least resistance.

i don't think there is a by attribute in the origin id

Ah, sorry that is only for stanza IDs. origin-id does not have by.

Peter is the author

That might almost always be correct but 0359 is Florian Schmaus ✏

> In addition, it SHOULD include an 'id' attribute that echoes the 'id' attribute of the content message.

And you want this SHOULD to be a MUST?

zinid, what some people here want is that origin-id = message-id

Ge0rG, you had an objection to that in https://mail.jabber.org/pipermail/standards/2017-September/033415.html but I don't quite understand it. Why do you want to know if somebody creates strong message-id?

SaltyBones: but the sender controls this itself, so what is a problem to set them equal?

zinid, we want wording that tells the client developer to do this

there is no 'problem'

SaltyBones: I thought it would matter, but it doesn't, because there always can be malicious entities

daniel: if they don't then they only harm themselves, I guess

zinid: no, they harm the other participants

Ge0rG: ah

How?

Yeah, how?

zinid: also, it's well possible that stanza ids are generated by the xmpp library, and origin ids by the client, causing a mismatch

By making all message references break

We had that above already.

Before we move to stanza-ids....

There is nothing wrong with requiring origin-id = message-id?

If we require this then why origin-id is needed, wtf?

zinid, it's possible that it was a mistake

daniel, does requiring origin-id = message-id actually solve your problem or are there cases when stanza-id might be used to refer to a message instead??

It solves my problem

daniel, wouldn't it be better to just mandate that clients always use the origin-id to avoid the problem of dealing with id-rewriting MUCs?

I mean, certainly removing an unnecessary id would also be desirable but this might be a good intermediate step.

Would everything be better if clients would generate proper stanza-ids?

For ...uh...some definition of proper that makes them UUIDish.

they should be strictly monotonically increasing, so we don't need XEP-0198

And by "would everything be better" I specifically also mean, wouldn't that allow us to ditch the other IDs?

and not UUIDs

zinid, "strictly monotonically increasing" is not gonna happen

because?

zinid, would increasing IDs really solve all the same problems as stream management? Seems unlikely?

zinid, also, increasing IDs have security implications

or rather: predictable IDs

SaltyBones, it solves in data replication, but of course XMPP is too unique

version vectors are based on such ids

jonasw, what security implications?

zinid, I think there were some IQ response injection attacks based on predictable IDs

even though in that case you probably already made the mistake of not verifying the sender of the response

error injections would most likely work though because errors can come from entities different than the original recipient

jonasw, this can be fixed by adding routing information

jonasw, maybe this is too much of an assumption for all of XMPP but don't we have transport encryption?

we anyway need this functionality already

SaltyBones, that doesn’t stop me (jonas@zombofant.net) from sending an iq type="error" id="whateverIguessed" to="you" to break whatever you were doing

zinid, I agree, if this is a problem I don't see why it cannot be exploited right now by somebody who randomly sees the message first

SaltyBones, if I can guess the ID, I can attack you from off-path

I can’t do that when I can’t guess the ID

if I’m on path, you’re right, everything is lost already in XMPP.

but if you cannot match incoming errors against requests you sent then you should really consider to change the job

zinid, how’d you match incoming errors against requests other than the ID?

you can’t really use from, because as I said, it might come from an entity you didn’t know about yet

jonasw, no?

jonasw, you can use (from, ID) I think

errors come from the original recipient that you addressed

MattJ, even if an s2s error causes an error?

Yes

oh okay

Otherwise it would be a nightmare

jonasw, There's a "by" attribute, if I remember right, that tells you the generator/reporter of the error.

So, I assume that if I run a h4xx0r server and try to send replies to messages that were not directed to me to some other server they will be discarded?

And the same happens between client and server...?

SaltyBones, by "replies", you mean error replies?

Whatever magical interferring replies that jonasw was referring to :)

yes, errors ;)

For that to work you would have to know the original sent message id, and the original sender's full JID

But if I did, I could?

and then yes, you could send them whatever you wanted - it would be up to their client to match it up (or not) to one of its outgoing messages

So, the statement that predictable IDs would allow me to spoof responses remains correct?

so as said above, the client should use (from, id) to identify error responses, not just the id

Because I cannot spoof "from"?

Correct

Yes, okay...

That's what I was looking for. :)

id "abc123" from userA@domainA is different from "abc123" from userB@domainB

And servers only accept s2s with from=...@domainA if they know that the other server is in charge of domainA?

SaltyBones, That's what they're supposed to do, yes.

^^

SaltyBones, yes, they do

MattJ, No, they don't, but I love your optimism.

If they don't, it's a bad bug or not an XMPP server

SaltyBones, So Metre (my S2S proxy mad thing) does this very strictly, and as a result I can't join this chatroom from my personal server.

SaltyBones, Not sure if MattJ is saying that's a bad bug in this server, or if he's saying it's not an XMPP server. :-)

Dave Cridland, for what reason?

Dave Cridland, so which messages do you get that don't pass?

Bunneh, version xmpp.org

SaltyBones: xmpp.org is running Prosody version 0.9.12 on Linux

MattJ, It's sending a response down the stream obtained by reversing the stream to/from, and not by reversing the stanza to/from. I forget the details, Zash knows.

MattJ, I don't know if Prosody would accept that or not. Metre doesn't, Openfire does (but because it knows it's multiplexed on the outbound, so weirdness.)

MattJ, For maximum fun, Openfire used to do similar "implicit" auth on outbound, but I stopped that as from 4.2.0.

I don't really understand, but maybe Zash will enlighten me

MattJ, Metre sends traffic for d.c.n -> muc.xmpp.org over a stream for d.c.n -> xmpp.org (after adding that domain via dialback).

How far back should I read to have any idea what you are talking about?

15:58 should suffice

SaltyBones, and a time machine

MattJ, Prosody (0.9) responds to the traffic on the reverse stream, xmpp.org -> d.c.n - which it hasn't requested authorization for yet.

Do you mean a timezone machine or a MAM machine?

No MAM here

SaltyBones, For your amusement and mild confusion, XMPP authorizes streams by a 3-tuple of (from-domain, to-domain, direction).

Page up button tho

SaltyBones, Well. One or more of those 3-tuples anyway.

Dave Cridland, who hasn't requested authorization for what? You mean Prosody connects to d.c.n as 'xmpp.org' and sends stanzas from 'muc.xmpp.org'?

MattJ, Yup, that.

The thing

MattJ, Like this: DEBUG 2018-02-12T15:14:35 /home/dwd/src/Metre/src/xmlstream.cc:95 : NS296914 - G ot [399] : <?xml version='1.0'?><stream:stream xmlns:db='jabber:server:dialback' xmlns:stream='http://etherx.jabber.org/streams' version='1.0' from='xmpp.org' t o='dave.cridland.net' xml:lang='en' xmlns='jabber:server'><iq id='472-452048' ty pe='error' to='dave.cridland.net' from='xsf@muc.xmpp.org/tim@boese-ban.de'><erro r type='cancel'><not-acceptable xmlns='urn:ietf:params:xml:ns:xmpp-stanzas'/></e rror></iq>

This time I'm joined. Timing, I guess.

Uh, what was the thing with that again?

Zash, The bug itself? Responding to inbound traffic on a multiplexed stream by reversing the wrong domain pair.

dwd, I don't see how timing would play a part

Dave Cridland: Do you remember where we discussed this?

MattJ, I think whether I can join is dependent on what streams are open, which is dependent on the session lifetime and activity.

MattJ, Not timing as in a race condition, mind.

Zash, Erm. 1:1 messages, I think, until we figured out it definitely wasn't a security issue.

Zash, Possibly some discussion in jdev.

FWIW, I have to put in something similar to an implicit auth for X2X, anyway. So this may provide a workaround.

MattJ: https://hg.prosody.im/trunk/file/0de0018bdf91/plugins/mod_s2s/mod_s2s.lua#l203

Zash, exactly where I ended up

stanza from/to may differ from the stream to/from in case of dialback multiplexing

MattJ, You switching to FreeBSD?

MattJ, https://svnweb.freebsd.org/base?view=revision&revision=329166

svn o_O

gitis too mainstream

capitalist pigs use git!

gititis

NetBSD has Lua in the kernel for years!

Dave Cridland: Heh, nice

https://www.phoronix.com/scan.php?page=news_item&px=MTMwMTU

And it has CVS :-)

damn, I only get used to svn...

Holger, Can you download NetBSD yet, or is it still only available on tape?

Holger, cool stuff btw

More importantly, once I download it can I run it on any machines made after 1995? (this is always the problem I had trying to run NetBSD; a few of the ops people at work run it, but they all use *very* old machines)

SamWhited, Sure! It runs pretty well even modern machines like the Amiga 4000.

(Sorry, that was 1993).

Exactly.

Well yes it's about as dead as XMPP :-P

I would like to use at *least* a P3.

But it always worked okay-ish for me on non-recent ThinkPads.

I no longer do that though.

cpu0 at mainbus0 apid 0: Intel(R) Core(TM) i3-2120T CPU

... is the one box I still run NetBSD on.

Uh, was NetBSD the one being difficult, or OpenBSD?

Difficult?

Actually, jokes aside, that might be a nice thing to do with my broken old thinkpad; doesn't work very well as a laptop anymore, but it could be a {Free,Net,Open}BSD or Illumos "desktop".

Zash: Isn't all computers difficult except for Apple?

This is true… ⤴

Et Apple, Holger

I find Apple incomprehensibly difficult to use, I must admit.

I've yet to figure out how to go up a directory consistently in file dialogs.

cd ..

Happy to help.

Managing FreeBSD feels more or less like Debian to me, the two others feel a bit more basic but not really harder to use. All dead simple compared to the dances you had to go through when partitioning hard disks or getting X11 to run 20 years ago with any Linux or BSD.

Holger, Ah, X11. Kids these days don't know they're born.

Kev, Don't get me started on the antiquated command line tools Apple foists upon you.

Yes, we're horribly old.

Holger, We're very experienced. I'm sure that's what you meant.

I'm sure wayland will bring back the good old times for those who miss configuring Xorg

Precisely.

The only thing I want out of a machine is to sync an old ipod without Rhythmbox crashing…

SamWhited, Rhythmbox never crashes.

SamWhited, At that speed it's called "parking".

Rhythmbox is one of those people who parallel parks by backing up until they hit the car behind them, then driving forward until they hit the car in front.

So, why is it that a client cannot create the stanza-id?

Is this a thing about malicious clients or what's going on there?

SaltyBones, no, unaware clients are sufficient; colliding IDs would lead to interesting issues with MAM

jonasw, but if the client is only unaware the server could just return an error informing the client that this ID has been used.

SaltyBones, But we could use, say, stream-id + attribute-id to form a stable, non-colliding reference identifier.

SaltyBones, that’d be annoying to handle in the client, and the server would need an O(1) (or similar) way to determine that the ID has been used already...

Yeah, or we could hash things and god knows what else...it seems very solvable...

SaltyBones, I mean, unless the client isn';t generating unique ids within its stream, in which case it's presumably not going to be fixed to use some other identifier.

Dave Cridland, stream id + sm-counter?

that’s verifiable by the server

and predictable for both

jonasw, True. But I think we had some ideas on predictability outside the stream?

throw a hash on top to avoid privacy issues, done

Dave Cridland, I thought we dicussed those into oblivion earlier? :)

SaltyBones, Really I've lost track.

Dave Cridland, hmmm ... hmac(stream-id, sm-counter)?

SaltyBones, I have the feeling that nobody quite knows the entire picture here.

I almost lost track and I basically didn't work at all today and just discuss here. :p

jonasw, HMAC() requires a secret to be of any use.

Dave Cridland, stream-id.

Seems reasonable on first sight...

(post-TLS obviously...)

jonasw, Is that secret? And why HMAC over a hash?

Oh, that would leave out the hash...

Dave Cridland, I’m fine with hash(stream-id || sm-counter) too, but that’s nearly an hmac ;-)

hmac basically IS a hash

yeah :)

jonasw, I'm pretty sure it's not. :-)

I am very sure it is. :)

that’s why I said "nearly"

I think the concat is slightly different

(yes, nearly)

https://wikimedia.org/api/rest_v1/media/math/render/svg/4edcf0bd8b403c93564b8d7ea91338b3208dea03

doesn’t matter anyways

jonasw, An HMAC is two nested hashed with concats and masks, yes.

Okay, so we only disagree on our definition of nearly. ;)

jonasw, But the security properties are distinct.

Dave Cridland, I don’t argue that

(and I’m aware)

Anyway, the suggestion of HMAC(key=stream-id, msg=sm-counter) seems good, doesn't it?

In any case, given a stanza with a known id on a given stream, we clearly want to be able to predict the MAM id.

I mean, I am also fine with SHA-*(stream-id || counter-sm)

Dave Cridland, but mam-id = stanza-id which is what I am proposing to set to this...

Question is, do we think the MAM id is likely to become public, and if so, can someone relatively easily figure out the next (or previous) value, and then do Something Bad?

Dave Cridland, that might be the question but if it only costs us a hash per message to not answer it, it might also not be... ✏

What I'm wondering, you see, is whether we give clients an algorithm to generate attribute-id and then let them signal to the server that they're going to use globally-unique attribute-ids and the server is allowed to call them out if they don't.

Dave Cridland, okay, right, this is about the MAM ID. this *probably* doesn’t matter, but if the goal is to consolidate all IDs at some point, choosing a way where the ID can become public is safer

jonasw, That's roughly my thought, yes.

and at that point, HMAC(…) seems like a sane choice.

jonasw, Well, that and "But we *HAVE* stanza ids - right there in the stanza!"

How would the server know that they didn't use a globally unique id?

Kev, if the server can simply re-run the generation algorithm and compare results..?

Kev, if the ID doesn’t follow the algorithm for a globally (predictable, for the server and client only) unique ID

Kev, Well, it'd know if they used no id at all, and if it saw a collision within a stream.

If we only cared about collisions within a stream, we could trivially solve all issues already.

And that is how you can tell that Kev is a VIP

when he says something three people reply!

Kev, Well, we care about collisions for a user's account for MAM, and no further.

But having a useful id to refer to a message would be jolly useful, and I don't see how we can get there yet.

Okay, so you're saying when two servers federate and you assume one of them is malicious how can the other make sure he doesn't get duplicate IDs?

Or non-malicious server with a malicious client.

SaltyBones, Do they care? ANd if not, should they?

SaltyBones, you don’t use other servers stanza-ids in your own MAM

Well, the malicious client would be detected by the server.

jonasw: Bloody useful if you could, though, no?

SaltyBones: How would the server know it's malicious, if it doesn't know all globally generated ids?

Kev, it can simply check if the client generates their IDs correctly...

Kev, Again, why would a server care?

Without exploding the size of IDs, at least.

(we should mandate the use of shakespeare-inspired adjectives in each bloody sentence in this room, by the way)

Kev, how would it be useful?

I'd like to send a reference to a stanza, where the reference makes sense in my archive, the MUC/MIX archive, and the user's archive.

jonasw, I don't think "bloody" is Shakespeare. "Submarine" is, though, as far as I remember.

Kev, you simply make it a hash or hmac of something that both the client and the server know and the server can check the calculation.

"simply" :)

Kev, but doesn't that only involve you, the muc/mix server and $other_client? i.e. only one server...?

Kev, Is the (from, id) attributes of the stanza sufficient? If not, why?

If this is for MAM purposes, then I will cry if I can't stick some time based bits into the ID

Dave Cridland: No, because the from gets rewritten.

Kev, When?

Zash, what?

In a MUC or a MIX.

My MAM ids are basically yyyy-mm-dd-random

Kev, those are different use-cases I think.

jonasw: If we cherry-pick the trivial use cases, it's going to be trivial to solve them :)

Kev, this is where origin-id makes more sense. It is generated by the client. If the client doesn’t ensure that there’s enough entropy in there, references to their message won’t work.

Zash, and this is important because?

Kev, Ah, so given a message from a MUC fanout, do you want to be able to identify the originating id? I'm not actually sure you always want to allow that.

I still don't get it. Why does from get rewritten?

And what does that have to do with anything? :D

SaltyBones, when you send a message from saltybones@yourdomain.example/client-resource to this MUC, we see the message from xsf@muc.xmpp.org/SaltyBones

so the from is being rewritten.

and if references are based on (from, id) pairs, you can’t use the same reference we all can use for your message

(simplified, of course we could still refer to the reflection of the message, but that would break down with MIX, too, I think)

jonasw: Well, no, because if you can fake someone else's id, suddenly you can maliciously change the target of a reference.

Kev, good point. so we indeed need something like (from, id) :(

Wait what?

lol

How can you now fake stuff again?

but given that all our fanouts and from-rewriting things do actually do reflections, I’m not sure if that’s so bad, Kev?

your local archive woudl still be able to resolve everything

jonasw: Well, it enforces a round-trip before you can refer to or correct anything, which isn't ideal.

Kev, you normally know the "from" you’ll be having.

and since you set the origin-id yourself, you already know everything you need

(meh, races with server-enforced nickname changes in MUC)

Point.

(but I guess those will be rare enough for us not to care)

halp! :(

SaltyBones: Because there are potentially malicious entities.

Are these only clients or servers as well?

jonasw: Still doesn't help where a user generates the same id twice, I think, and you're left with ambiguous references/corrections/whatever.

SaltyBones: Both.

So why do you connect to a malicious server and expect things to work?

Maybe that's not a good approach.

Kev, that’s not an issue. put 256 bits of entropy in there and you are officially allowed to not care

Ah, if you have a mechanism for identifying malicious intent on the Internet, you could be a very famous person.

Kev, and if we want to have defined behaviour in that case, always assume the most-recent message

jonasw: That only works for accidents, rather than manipulation, no?

Kev, but you cannot manipulate as long as the server is checking.

Kev, indeed

but if you manipulate your own origin-ids, it’s your own fault?

Of course if we have malicious servers now a bit more work is required. :)

And I'm concerned that 'most recent' falls apart if you can manage different people receiving different subsets.

can one inflict harm by producing duplicate origin-ids?

I’m not convinced that this is actually an issue.

Most likely.

If I can cause the 'wrong' message to get corrected.

maybe with something like Reactions based on (from, origin-id) references.

Or a like to apply to the wrong message.

If I can manipulate you into liking a Godwin instead of something sensible, that's not ideal.

meh

But messages aren't authenticated anyway, if you are a malicious server you can claim to have received whatever likes from me...

Kev, then the only way is a MUC/MIX stanza ID generated by the MUC/MIX and waiting for a round-trip before references can be done.

SaltyBones: Other way around.

SaltyBones, all it needs for now is a malicious client though

SaltyBones, or a malicious client+server pair if we do the hmac-stanza-id thing for origin-id

I can still run my own server and make it fake origin-ids

your own malicious server?

Kev, I’d however argue that actively, from a client/own server side, make participants in a MUC/MIX receive only parts of the conversation would be rather trikcy

and targeted parts even

and that without them suspecting that things are broken in funny ways and thus not trusting the system anymore

jonasw: Ah, that's ok then, no exploits have ever involved doing anything tricky :)

Kev, I see your point, but I’m not sure this is actually something which is reasonably exploitable.

but yes

unless we let the ID be generated by the fanout service, there’s no way to be sure I’m afraid

I mean, I don’t have an issue with that, I’m fine with that even.

Yes. I don't see a way of solving this, which was why I brushed it under the carpet at the Summit.

complicates things though

Kev, I’m having a really hard time constructing a successful attack which wouldn’t be seen by the victim in the MUC case

even when I omit arbitrary messages

to only some participants

(which would be really really tricky to achieve targetedly I think)

ah, now I have it

this is the scenario: 1. user A, "bad statement", origin-id=1 2. user A, "good statement", origin-id=1 3. user B, like (from=user A, origin-id=1) if (2) isn’t seen by all users, they see user B liking "bad statement" instead of "good statement"

with an arbitrary amount of messages between (2) and (3), it is also not too difficult to make people not see (2) in a MAM-less MUC.

Indeed.

I don’t think there’s a way unless the MUC does things there

But, all it requires to prevent that is for the MUC to check that the ID is unique just like the assumed-non-malicious server did in our previous discussionn...

(i.e. if the MUC generates the ID) ✏

SaltyBones, but that is a rather expensive check to do

you need to record IDs for all eternity, or have a defined way to generate the ID which can be executed by clients

the latter would be tremendously tricky to get to work synchronously

but possible...

Which is a little more tricky because there is no obvious common "secret" like the stream-id but a simple Hash("counter") would do

something like hash(message-counter || presence-id), where presence-id is an ID assigned to the client on join

Kev, note that in this case I used the word simple again but I was totally lying... ✏

hash counter doesn’t really work with multi-session nicks I think, it would lead to collisions or races

jonasw, yeah it was just to get the idea. Indeed you would at least need some salt.

But if the server gives you the salt on join and you use that to generate the IDs it should be good again.

that would be verifable by the MUC and the MUC would drop stanzas which do not adhere to this schema

(or rather, reject)

and since the counter can be per-muc there is no issue with having it

per client even

I think :p

Kev, ^

well, yes per client and per muc

just saying it doesn't seem to be a privacy issue

SaltyBones, yeah

so origin-id = one-way-function(presence-id, message-counter), where presence-id is assigned on MUC join

only need to define what happens when message-counter wraps around or becomes large or something

(same thing for SM by the way, SM has wraparound semantics)

O_o

Y U SEND SO MANY MESSAGES!?!!

SaltyBones, itym "y u have so long sessions"

still, if this is defined somewhere we can probably renegotiate the salt at that point...

while wrapping around a 64bit counter in a single session is sure challenging, we need to be prepared if this is to be solid :)

yeah

I hear counter?

sorry, I don't track the discussion

zinid, I’m not going to repeat everything you can read in the backlog :)

as you wish

not that I care

counter works in this case because you can restart counting when rejoining the muc

whatever you will end up will be shit anyways, so

you already created 4 fucking ids: stanza-id, origin-id, attribute-id and SM id

maybe it's time to stop and think?

what is an sm ID?

h

right

zinid, all of this is about reducing the number of IDs, so I thnik this is kinda what we’re doing?

jonasw, from what I see you want to add yet another counter

zinid, to generate origin-ids, yes

for MUCs and MIXes

to make them verifiable by the service

zinid, or rather, we’re replacing origin-id by some one-way-function(counter)

how this will cover sm id?

We haven't discussed sm-id, yet. Only the other three.

I actually have no clue what sm-id is. :)

SaltyBones, then I need to wait while you recognize the problem 😉

zinid, so reducing 4 -> 2 is not enough, eh? :p

zinid, sm-id stays, but stanza-id (and attribute-id) could potentially become one-way-function(stream-id, sm-id)

2 is enough, however I think first should be counter and second should be routing information, and not an ID

stanza-id being used for the archive only, origin-id being used throughout the network together with the sender address for refernecing a specific message

what ID will be used to sync?

with SM IDs you just create a pointless queue in c2s state

Okay, I think we are trying to solve a problem that is very orthogonal to stream management. But we have only discussed this a bit and it might very well not work even for what we want it to do.

SaltyBones, I don't think SM should be separated from archive

it's pointless

Why?

zinid, stanza-id is used for ysnc

(for MAM sync that is)

SaltyBones, because why you need this separation?

SaltyBones, you keep messages in MAM and then put them in c2s SM queue

why can't you just inc(counter), store in MAM and send via c2s?

Why are you asking me that? I am 100 % sure that you know more about SM than I do!

what I know about SM is that I need to maintain stupid queues inside c2s processes, which sucks as hell

even though a client can request those messages via MAM

if you receive an ID out of order, you just reconnect and ask everything started from the ID you received

and server will send you this from archive

and vice versa

But SM-IDs are per-stream not per-message, right?

At least that's what it looks like in https://xmpp.org/extensions/xep-0198.html

when you connect a server, you provide last seen ID and it will resend you everyting what's greater than this ID

SaltyBones, yes, they are totally separated instances

What if the thing you missed was an IQ or a presence that isn't stored in MAM?

If you temporarily disconnect and miss something, SM acks allow you to find out. Doesn't help as much if it only covers messages.

SamWhited, I think we can drop them

SamWhited, we already don't care about IQs with Push, so why would we start care?

try to make jingle call when I'm in "push" mode

Does that not work? That does seem like something we need to care about to me

SamWhited, but we don't and we cant with push stuff

shouldn’t an IQ trigger a push? :-O

SamWhited, what if I want to receive your software version and you're in push mode?

That seems like a problem that needs fixing then, not an example of something being done right that we should copy.

anyway, you can keep IQs in MAM if you prefer

you need to keep subscription requests for sure there

we keep them already, but in a separate database due to historic reasons only

IQs (which are inherent request-response, with exactly one response) in MAM sounds like a terrible idea.

yah, now you have to try and figure out which IQs are time sensitive, which are important to store, etc. it seems like a comlicated way to work around having a stanza counter.

SamWhited, but you have to decide now too: for example, after 5 minutes of inactivity (by default) a server just bounce all IQs from SM queue

*bounces

I'm not saying it's perfect, just that this doesn't seem like a good fix.

a fix? the behaviour is the same

but we let a client to decide which IQ to reply

Except now you have to store all stanzas in MAM, or not store some stanzas and risk those being the ones that are dropped and you don't know you missed something. The point of SM is to make sure you know if you lose something, which is often important.

do you really want to loose incoming call?

that's how it works now: if somebody calls you and you're not connected you loose any track

As I said, I'm not claiming that the current solution is good; just that this makes it worse.

no, it makes it better

you just need to define what to store and what not, well, it's too much to redesign, yes

If you don't store everything you now lose the ability to detect connection drops though

sigh

not sure how will you address missing call though

with your approach

I do not have an approach; I agree that is a problem that needs solving though.

It seems like we need storage for management messages and storage for actual messages. I don't see why these should be mixed up.

we need to decide what to store and what not

hm..that distinction was wrong, yeah

there should be a storage until delivered and an optional long term storage

users might not even want mam :p

if you don't want mam then just lose messages

Okay, let me rephrase, some users might not want long-term storage of messages...

not sure why this should be specific to any approach

Because I think that's what MAM is intended for and that's why using it for all messages as you suggest is strange and might have unexpected results if people built it under the assumption that it is for long term storage...

servers can drop MAM archives on reconnect for example, what's the problem?

Okay...

or later, when it's delivered

this is just me confusing MAM and the other thing again...

for the one-millionth time...

What if I want my messages both on my pc and my mobile? I can't just drop MAM when my pc is online

you can if everything is delivered

anyway, what you are trying to say is a partial replication, and this problem is very hard to resolve

what is the other thing again, server side archiving?

zinid, does MAM know if everything was delivered?

SaltyBones, well, no, I think you cannot know that in general case

zinid, but that is what SM does, right?

SaltyBones, well, if you introduce acks, then yes

Hm.... :)

All this is orthogonal to the question whether having two separate stanza/message queues is sane. I agree with zinid that it isn't.

Two whatnow

I'm saying for many years now that we need to replace 0184, 0198, 0280 and 0313 with one single proper message syncing thing.

SM is already mostly just an optimization, and I think we should fix the remaining issues to make stream management superfluous.

I just want to point out that this is orthogonal to our earlier discussion about IDs :)

SaltyBones, I said "no" in general case because we have Bysantine General Problem, it's unresolvable, what we can gurarantee is sequential consistency

SaltyBones: please show your ID before being allowed into this room :P

Holger, does anybody already know what those issues are?

Ge0rG: Kidnap some server devs and lock yourself in a room until that one single proper message syncing thing to rule them all is properly implemented and XEP'd

SaltyBones: Syncing of outgoing messages (in a sane way) and maybe avoiding some round trips during session startup.

Zash, and watch them die 😉

Zash: I'll lock you and zinid up in that room, and watch the live stream

Holger, so, can we just turn off stream management and leave MAM and see what happens to find out?

Sure you can :-)

Or do you think fixing up MAM is a bad idea and something new is required?

SaltyBones: That's what I did, actually. Haven't died from SM-lessness yet.

SaltyBones: I think you can already implement proper sync with MAM as-is. But something new is required to let clients implement this in a sane way, without having to de-duplicate and whatnot.

that's my point: what we need is to store messages and some other restricted stuff and call it a day

Holger, so would unique message IDs solve this?

I mean, at least de-duplication would be reasonably easy then, I guess.

Of course some sort of counter makes more sense in this case...

SM has counters. MAM has server-issued, guarante to be unique message ids.

Why are IDs not a counter?

Not sure what sort of uniqueness we need to solve what problem. Didn't read the backlog, sorry :-) The thing missing for proper sync of outgoing messages is an algorithm to compute the MAM IDs of outgoing messages.

I mean the MAM-IDs....or is that the same as the stanza-ids?

(Sorry, typed this before reading the last few messages.)

Zash, now answer the question "get me last messages I didn't receive" with current SM and MAM approach 😉

Holger, what do you mean by outgoing? Messages that we sent?

Yes.

if you say "time", then no

zinid: query after = id of last message I saw

Okay, that should be solved by the hash idea....if it works :p

cry over outgoing messages sent after that

SamWhited: I need to know their IDs so I can tell the server "give me all messages since $id".

But indeed if we want to query MAM by a "point in time" or "counter" unique IDs are not really the best thing :)

Holger, but then the server still has to search the MAM archive for that ID and give you everything after it....

Zash, define after in distributed system 😉

So a counter would be much better.

SaltyBones: Sure?

zinid: MAM archive ids on incoming messages

Right?

zinid: after is a MAM term

Zash, but you need some ordering

zinid: huh

zinid: XMPP streams are ordered

yes, but you need to maintain a timestamp index in the database

You've lost me

well, you're probably right that if we assume timestamp ordering then we don't need counters and SM at all

Huh?

In a MAM query, 'after' is a field that carries a MAM archive ID

so, what's the ordering? 🙂

ID+1?

from what I understand you use timestamp+id, which means time ordering

Ordering?

As I said, you lost me. I have no idea what any of us are talking about anymore.

ok

https://summerofcode.withgoogle.com/organizations/?sp-page=5 no xsf?

BEAM community is there 😛

SaltyBones: like 98% of what you were talking about is here https://github.com/moparisthebest/xmpp-ircd

Basically it works if IRC users don't want nickserv or chanserv but they do and I never got back to it :)

moparisthebest, you mean of a way to connect to a muc using the irc protocol?

Yes

Running that makes a muc look like an IRC server to an IRC client

I'm pretty sure nobody wants that. At least I don't! :D

But I think it might provide an excuse for people who have to convince irc users ;)

SaltyBones: yesterday you said

Hm...maybe the transport should be the other way round. Offer an IRC server that connects to MUCs.

That's what I was referring to

Ignite didnt make it into GSoC

:(

moparisthebest, I know, I said that regarding the discussion of the KDE folks looking for an IM solution

`> * Ge0rG had a very concerning realization about guessable IDs and packet filters in smack earlier today Care to share?

Or is it just something in the ancient smack library yaxim uses?

SaltyBones, right so they could use a MUC, but also have an IRC server that IRC users could use

moparisthebest, I like this, BTW.

and everyone would end up in the same place, but it'd be a muc

moparisthebest, Is the GPLv3 your addition or from telepaatti?

Anyone know what servers support XEP-0288 these days?

Does this Prosody instance, for example?

Dave Cridland, looks like the original telepaatti is gone, but GPLv3 goes back to at least the next fork looks like

honestly I kind of abandoned it because I like rust so much more than python nowadays but can't be asked to rewrite everything yet haha :)

-xep 288

Flow: Bidirectional Server-to-Server Connections (Standards Track, Draft, 2016-10-17) See: https://xmpp.org/extensions/xep-0288.html

I still run an IRC server I would love to rm -rf

> I need to know their IDs so I can tell the server "give me all messages since $id". I know, apparently I wasn't clear about something, sorry. I'm not suggesting we get rid of MAM or leave everything exactly as it is today, just that MAM doesn't cover some important parts of SM and probably can't be made to cover it without significant downsides.

Flow: it's affecting the old smack for sure, I'll have a look into smack 4 and let you know

(sorry for the long delay, got pulled into a meeting and then was AFK for a bit)

So there's actually only one server I talk to that does bidi. That's scary.

Ge0rG, let me know if you didn't just re-discover CVE-2014-0364

Flow: it's related but different

Dave Cridland: Is it mine? :)

Zash, No, Lance's. Although actually grep might have gone into Annoying Binary Mode.

-I ?

-a, but yeah.

OK, so that's actually lots of servers doing bidi. Happy bunny, now. I've been putting the support into Metre.

Oh? Hm, feeling up for doing a survey?

Zash, What sort of survey?

"How many servers do 288?"

Hooooold on now

Is that the same number as ...

https://www.youtube.com/watch?v=azEvfD4C6ow !!!

Zash, Looks like it's PSYC and Prosody.

Prosody doesn't do it out of the box, you need to go a bit out of your way to install a community module.

Really? Quite a few people have, then.

Dave Cridland: Question is, how much self-selection bias is there among people that have you in their roster? :)

Zash, Lots.