XSF Discussion - 2019-09-10

pep., ask a vegetarian

ralphm, if I am not mistaken, the current rules of rfc7622 disallow unassigned to in resourceparts, domainparts and probably also localparts

i'd say the spec is sound and as sensible as possible, it is the implementations that do not follow the rules and so, once in a while, an invalid jid slips through. That's the main motivation for creating the jid/xmpp strings testframework and the valid/invalid jid corpus

flow, except that RFC 7622 does not pin the unicode version

so one entity running on Unicode 10 could consider something as legitimate which an entity on Unicode 9 would not

jonas’, right, but as I said earlier, I would consider this to be very rare. But I could be wrong. And I don't think there is a better solution, happy to be proven wrong though

That is, I think the tradeoff of not pinning the unicode version is justified

At least the troubles we had so far are not caused by not pinning the unicode version, as far as i can tell

No, but they have the same symptoms

Isn't my example a sign of why this is a problem? Emoji are all Symbols (So), I believe, and as such valid in parts of JIDs. Differing Unicode versions have different ideas on newer codepoints, so also on validity of JIDs?

If we don't want to break the experience for everybody when somebody employs new unicode, we need to accept unassigned as valid from remote entities

The problem with that, though, is unassigneds that become prohibited.

Like U+061C.

Since the foremost expert on this is Peter, I suggest someone write an email about this to standards@. He's busy, but it's more likely he can respond there.

I'm not sure he'll be able to solve that problem either ;)

No, but he can at least confirm we have this problem and/or know about strategies.

07:07:12 Ge0rG> If we don't want to break the experience for everybody when somebody employs new unicode, we need to accept unassigned as valid from remote entities

that’s only a partial solution

codepoints may change categories and stuff between unicode versions

and an unassigned codepoint in one version may well be a RTL-codepoint in another version

so by accepting unassigned input, you may accept something which someone else will consider invalid.

unicode is a mess.

ah, ralphm said that already ✏

well, since the problem is mostly in resourceparts, localparts and domainparts forbid emojis, we should probably establish a pattern that resourceparts are not user-configurable nor user-visible. Shame on you xep45! I wonder what the state in MIX is

And we should probably add a note to xep45 that the use of certain unicode categories is discouraged

But I don't want to be the person to discourage emojis in muc usernames…

flow, passwords and such are also affected.

jonas’, how's that?

localparts can be Emoji as well.

flow, passwords are also passed through stringprep/precis

Ge0rG, localparts are UsernmaeCaseMapped profile of the IdentifierClass, and that class forbids symbols under which emojis fall, no?

Maybe not all emojis, haven't check them all

In MIX, nicks are an attribute of a participant, not part of their identity. However, it also says you have to follow https://tools.ietf.org/html/rfc7700

Which in turn depends on Precis FreeformClass, and thus has the same issues as resources.

guess users just want emojis in their nickname

maybe there is a reserved for future emojis unicode range?

there’s still the problem that you can’t do proper normalisation if you don’t know the codepoints

well if the reserved range also states the properties of the eventually assigned codepoints?

that won’t work

then they could just be assigned

stuff like how they combine with fitzpatrick modifiers

No because you don't now yet what they are assigned to

but if this codepoint is assigned to, then it has the following properties

btw, there is an excellent post about this topic at https://hsivonen.fi/string-length/

flow: I have a user ♥@ツ.op-co.de

Ge0rG, I am not suprised that you do, if that's the question

flow: no, when we think of as emoji is all over the place in several Unicode blocks.

ralphm, I suspected that to be the case

flow: I'm not aware of it being illegal. ✏

http://www.unicode.org/charts/PDF/Unicode-12.0/

Anyhow, yes the situation is not perfect, and I am happy if we could improve it. I just don't know how, and I can probably live with the status quo

I like the one on chess symbols: https://www.unicode.org/charts/PDF/Unicode-12.0/U120-1FA00.pdf

Actually https://tools.ietf.org/html/rfc7564#section-12.3 spells out the issue quite clearly: “Strings that conform to the FreeformClass and many profiles thereof can include virtually any Unicode character. This makes the FreeformClass quite expressive, but also problematic from the perspective of possible user confusion. Protocol designers are hereby warned that the FreeformClass contains code points they might not understand, and are encouraged to profile the IdentifierClass wherever feasible; however, if an application protocol requires more code points than are allowed by the IdentifierClass, protocol designers are encouraged to define a profile of the FreeformClass that restricts the allowable code points as tightly as possible.”

(there's a similar remark in the interop section 13.

)

*sigh*

sad that the emoji which could express my feelings right now is only coming in unicode 13: Smiling Face With Tear

But is the situation really that bad? Implementation could get the latest unicode standard over some sort of data network once in a while. You don't even have to update the involved libraries etc.

flow, is that true?

I think that highly depends on the libraries

I’m not sure how to update python unicodedata for example without updating python

There are libraries that still do just resourceprep instead of Precis, simply because RFC 6122 is directly linked from RFC 6120, even though it is obsoleted by RFC 7622.

One example is Twisted, which I am author of.

One could argue that with resourceprep being more restrictive, just having that is at least a bit clearer as an interop goal.

To be honest, I don't know what the best course of action is in this regard.

stay with unicode 3.2 forever

ralphm: be liberal in what you accept and strict in what you emit

s/emit/allow users to send/

would a MUC service be strict or liberal, regarding nicknames for example? :)

Ge0rG: my argument here is that this means that something like U+-061C causes problems.

Zash: yes, I implied that

It was unassigned before (so not valid), then assigned (but still invalid).

But 🥓 was unassigned before (so not valid), and now assigned (but valid)

ralphm: yes, but if the MUC service accepts it, other servers or clients receiving it from the MUC shouldn't freak out

i.e. a MUC service can strictly police the nickname, but not the resourcepart of the users' real JID.

ralphm, it’s not invalid, it’s only invalid if used with LTR characters :)

A MUC service is not something magical. It is just another server that connects to other servers over s2s and uses JIDs in addressing of stanzas.

jonas’: it is invalid as it is a control character.

ralphm: a regular server should police the resourcepart of local users, but not of remote users.

jonas’: (for FreeformClass)

ralphm, ah, fun

Ge0rG: well, that might be sensible approach, indeed. I'm not sure how well that works with mapping on new code points, and what kind of normalization issues arrise from that, but ok.

In any case it deserves some wider attention. Maybe even to the XMPPWG mailing list.

ralphm: framed differently: you shouldn't police any JIDs that you don't have the authority over, except when they are illegal in a breaking way, i.e. contain " or '

does that include localpart?

ralphm: what?

Ge0rG: should a server do precis processing on localparts of a remote JID?

Ge0rG: also, for resourcepart, should it a) use incoming JIDs as is (no processing), b) allow unassigneds, but still do Precis, c) something else.

ralphm: I'm not sure yet where the point of no return between a and b is, for either localpart or resourcepart

If you do a, that probably opens up some very interesting ways to break your clients

I think it boils down to: treat JIDs as opaque if you don’t have authority over them

Yep, things like IV and Ⅳ.

don’t do normalisation on them, or any processing at all, just treat them as opaque sequences of codepoints

(I followed by V, vs. ROMAN NUMBER 4)

ralphm: I don't think _that_ would break things

it is the domain authorities responsibility to ensure that stuff is valid and comparable when it is emitted from there

jonas’, I think so. You sure could bulid an python library that does so

but you can then have different people with arguably the same nick

ralphm: this is something the MUC has authority over.

ralphm: if you try to enforce that on your user's server, your user will get kicked

Right

> jonas’> ralphm, it’s not invalid, it’s only invalid if used with LTR characters :) I think it is invalid regardless the context with rfc7622

But I definitely don't want to be so lenient for localpart

ralphm, why?

ralphm: just tear down s2s and blacklist the remote server as incompliant.

flow: it is invalid in resourceprep because unassigned in 3.2, and invalid in Precis FreeformClass because it is an a prohibited class

Conveniently, it also prevents you from contacting the server admin

jonas’: because (bare) JIDs are identity

ralphm, from whose perspective are you currently arguing?

jonas’: I don't want to accept incoming stanzas that fail precis processing on localpart

as a client? as a MUC service? as a server? as anyone?

all, I guess

I see

> jonas’> don’t do normalisation on them, or any processing at all, just treat them as opaque sequences of codepoints That would probably open up another box of issues

Since Unicode does us so much good, I'l like to suggest that the XSF adopts a character (for as little as 100$, but maybe we could got for silver) before matrix does it: https://www.unicode.org/consortium/adopted-characters.html

+1

flow, send this to board

on my way

and find a good character thing to sponsor

U+1F5E9 probably

but I am open for suggestions

I propose U+1F926

💡 U+1F4A1 would be too obvious, right?

uhh

Would be nice to havethe logo as a character :D

Ge0rG, nope :(

flow's suggestion makes more sense ;)

Seve: that can only mean you are too young.

Seve, https://www.jabber.org/

Nah, but I want to go forward!

;)

Gaze at our bright future, my friends!

https://upload.yax.im/upload/8O5TitoHucjZZDeW/Screenshot_20190910-111700_Firefox.jpg

same here

we’re the future!

(a.k.a. WTF)

Looks rather like a SEMI OPAQUE RECTANGLE

flow: Discourse already has Gold on U+1F4AC, so yeah.

To be honest, funny as it is, I don't think we should spend any money on this.

What's the conclusion of all this btw?

(Not the Unicode sponsoring bits)

pep., everything is terrible

I think the most sensible statement is around 08:38:12 ralphm> In any case it deserves some wider attention. Maybe even to the XMPPWG mailing list.

Can somebody(tm) put that to the agenda if they think it's appropriate?

So that we don't get stuck here and realize we still have the same issues in 4 years

Gotta have this discussion every 4 years

hmm, I wonder if there is a backstory behind the pile of poo gold sponsor: https://www.unicode.org/consortium/adopted-characters.html

I'd like to think that friends of Jason raised the money and did this behind his back.

Maybe that name is a kind of pseudonym with a secondary meaning?

Random quote found through google: "that's a shitty way to spend 5000 USD"

I suppose there are enough rich brogrammers in the valley

For those involved in the Unicode discussion: I wrote to the XMPPWG mailinglist: https://mailarchive.ietf.org/arch/msg/xmpp/a-WhzOTyOq168GujQHgzQ1-DURI

thanks

<3 thanks

where do I subscribe?

https://www.ietf.org/mailman/listinfo/xmpp

thanks ralphm!

and beware IETF Note Well https://www.ietf.org/about/note-well/

"there are implementations and deployments performing the obsoleted stringprep." you mean all (at least public) implementations? :P

I raised this sooooo long ago (back when we were discussing using precis for JIDs in the first place).

The opinion then, as I remember it, was mostly to not worry about it and assume it won't cause practical interop problems that people might be talking different versions of unicode.

given that we had a fun unicode version interop problem the other day, I think we can safely bury that assumption

That's ok, I didn't believe it at the time :)

good :)

:-D

🤖 will disagree on that

that is also PRECISely my problem with it.

someone had to say this, and now it’s out of the way, you can all thank me.

;)

🤦‍♂️

Kev: I guess that was all before we got gazillions of emoji that are valid in resources.

Yeah, somebody hijacked the Unicode consortium to do things actually relevant to the bigger populace

𒈜

where was this repository where Daniel explains how the push service for Conversations works and which data is passed to google exactly?

ah, found it

https://github.com/iNPUTmice/p2

Cool story bro

> 𒈜 What was that

😉

Hmm, I'm missing the message to which Zash responded "cool story bro"

https://igniterealtime.org:443/httpfileupload/5c99fd39-7a01-40ab-8da9-b3e97d387824/rnGY3VwZTG6XbONXbZUg_g.jpg

Odd, it's in Dino but not poezio

I saw it in Converse, not Conversations.

More unicode magic?

indeed I don't see it.

Something something message dedup?

It's only in Converse that I noticed the "I am groot" message.

I already wondered why Zash was reacting with that on the message that I saw before it.

Guus: me too

Now I want to see the xml

Unsure if it's in MAM

It's not in the MUC MAM

Ok, what trickery is this

Can anybody post the XML?

and I was wondering why Zash thought my finding of the p2 repository was a cool stoyr

Can't post the XML. Can't even find the corresponding line in my logs.

it's not in my dino

it showed up in my dino

It's a carbon.

a carbon in a muc?

https://burtrum.org/up/7fa35ad6-3c2e-4f19-b0a2-acb54255d6ee/open-screeny-16761.png

<message to="georg@yax.im/poezio" id="718d40df-3948-4798-a99b-35cc9f03cc4f-641" type="groupchat" from="xsf@muc.xmpp.org/balu_der_baer"> <received xmlns="urn:xmpp:carbons:2"> <forwarded xmlns="urn:xmpp:forward:0"> <message xmlns="jabber:client" to="xsf@muc.xmpp.org" type="groupchat" from="xsf@muc.xmpp.org/i_am_groot"> <body>I am groot.</body> </message> </forwarded> </received> </message>

so any client that shows it potentially has f'uped carbon parsing?

Royally

yep missing from my Conversations though, neat

I love that mysterious bug finder

Ge0rG, do you just dump all the xml?

Daniel: that's from poezio debug log file

Everything old is new again. https://www.cvedetails.com/cve/CVE-2017-5589/

sadly i think dino even existed back then

It's interesting to ponder on how this can be utilized to have covert discussions en plein public

moparisthebest: Guus: can you open bug reports?

Daniel, but you said it *didn't* display in your dino? but it did in mine... what version do you have?

Guus, MUC PMs seems simpler

HEAD

but maybe it wasn’t stored in muc history

Zash: where's the fun in that though

so don’t count on that

AH that makes more sense

Ge0rG: wilco

mine is built from git HEAD too, but trying to figure out exactly when...

Also I need to talk to our content manager because the advisory url is 404

Mine is whatever Debian package from OBS, and I saw it.

jcbrand: ^^

converse showed it as well?

Funny how the month changed... https://rt-solutions.de/en/2017/01/cve-2017-5589_xmpp_carbons/

sigh

Converse was affected back then.

balu_der_baer: are you a pentester or is your client broken?

that does not look like a broken client

(on the sending end)

Daniel: something like delayed delivery gone very much wrong?

how? why?

Next up: unrequested MAM impersonation

the `i_am_groot` seems like a dead giveaway for deliberate test

otherwise that'd be an insanely odd client bug

there is so much long hanging fruit to pick in the xmpp world

It's good that somebody does the testing. And this place is actually well suited

So what's next, shall we try the MEGALOL-attack?

It would have been nice to share findings though.

I found out by accident.

isn't that what that was? :D

i mean i was wondering why Zash found the p2 story so interesting…

Daniel, same :D

Heh.

"complain loudly if you can read this"

haha

so you can probably impersonate actual people that are in the MUC right?

moparisthebest: yes

depending on how fucked it is not just muc

moparisthebest: most probably you can impersonate anyone, even outside of the MUC

moparisthebest: read the CVE

right, sweet

yea I just meant the XML groot just sent was MUC only, and implied you could impersonate anyone

I'd seen the old general carbons CVE before though

It's not really new

We should have a test suite for clients.

i wouldn’t be shocked if dino was vulnerable to CVE-2015-8688

https://wiki.xmpp.org/web/Client_Test_Cases

so is this covered by this line in the XEP

someone should try; probably...

Any forwarded copies received by a Carbons-enabled client MUST be from that user's bare JID

?

lovetox, yes

someone cant fake a message from a bare muc jid

Uff, this was hard on mobile. https://github.com/conversejs/converse.js/issues/1704

Please augment if needed

lovetox, it not bare jid. just the users bare jid is allowed

there shouldn’t be carbons in mucs

yeah but the server is responsible that there are none

at least that says the xep

huh?

your carbons parsing code needs to be wrapped in a if from == null || from == my_account_jid

ah i get it

yes must be from my account bare jid

not a "user"

which excludes the shit balu send

yes

# Carbon must be from our bare jid if not stanza.getFrom() == own_jid.getBare(): raise InvalidFrom('Invalid from: %s' % stanza.getAttr('from'))

was scared i fucked up :) but seems i did this right

That's not a new bug, gajim would have probably been tested at that time :)

I've added a section to the test cases

thanks

Still looking for somebody who can implement them

Would probably have to be a component for the MUC parts

OTOH, a bot could fake being a MUC, right?

yes pep. but as of course i think i can do everything better i reimplement much code, also carbon parsing

This carbons thing could be done by a bot

hehe

lovetox, tests!

It was a huge strain to my eyes, my fingers and my patience to add those three lines to the wiki from my android phone.

though its much harder wth MAM

i only accept mam messages with query-id s that im actually waiting for

well you do…

and yes can confirm that dino is vuln to https://gultsch.de/gajim_roster_push_and_message_interception.html

why does this shit keep happening

#BSG

BSG!

BSG?

so question is do i fix it now?

Daniel: can you do a roster push through a MUC?

Ge0rG: looking at the code I'm relatively certain you could

Yay.

let's try?

Haven't tested that one tho

You have to get lucky to get your iq routed I guess. Lol

Daniel: only with MSN

is there a generic bot/component someplace that can just try all of these things against a JID

Which is probably the default in this MUC

So not a correct target

so it can be used across projects

moparisthebest: write one please! https://wiki.xmpp.org/web/Client_Test_Cases#Staying_inside

it would probably be hard to write it with most existing libraries, they tend to try to insist on you sending proper things

Glad the Spammer haven't found out how to but themselves right into your roster

The cool thing about that CVE is due to roster version it also won't go away

I'd gladly accept spam from such a smart spammer though

So my Dino will be stuck with that test jid I injected

might even buy what he's selling

moparisthebest: it would get propagated into the spam sending tools and used by dozens spammers within some weeks

So who is going to collect the CVE for mam injection in multiple clients?

Daniel: let's wait half a year until there is a significant deployed base

🔥

Other than that, I'll gladly volunteer. I need some more CVEs on my CV

CVEs go on your CV?

Zash: yes

thats why they start with CV..

:D

Curriculum Vitae Extension.

Do we have an up to date entity caps database?

Can you see me?

Only the hash? Or all features? If it's just hashes, movim probably has a few up to some point in the past(?) https://nl.movim.eu/?about#caps_widget_tab, otherwise I'm sure you can gather some by running code on prosody

balu_der_baer, yes

A wild haxxor appears

balu_der_baer: no

pep.: all the features. Looking for clients with MAM

Mam doesn't show up in Caps

Then I'll hack something into mod_mam

Shouldn't show up. 😂 ✏

Nothing says you can't do client-to-client MAM ;)

Zash: MAM Push!

Idea from long ago: Make a bot that connects to your account and enables carbons, then lets you query it.

Zash: that's actually been mentioned a few times..

Like posting some Carbons when upgrading from 1:1 to a private MUC!

(c2c MAM)

There used to be an ad hoc command that did something like that

pep.: nothing new under the sun.

Only for unread I believe

Yeah, that too

from reading the code it looks like dino has disabled code that would have checked for the origin of a mam message

and yes it is in fact vulnerable

(just wanted to beat Ge0rG to it)

Daniel: a worthy goal.

I don't want to swear that slix isn't.

(or poezio)

vulnerable to what?

to the MAM thing? no i bet it will be more than just dino

we don’t check the origin, but you have to guess the (fully-random) mam ID

(slix matchers make checking for multiple things a bit tricky, so to fix that we would have to write an "xml mask")

Daniel: keep us updated on your advisory

so this time balu_der_baer 's "Can you see me?" showed up in Conversations but not dino, fun stuff

was that anything critical?

Ge0rG, got raw XML for that one?

04:48:04 IN <message xml:lang="en" from="xsf@muc.xmpp.org/Daniel" type="groupchat" to="pep@bouah.net/poezio-C7iY" id="e682bdd7-d98c-4cfd-9c59-fb9e5f9a6d8a"><origin-id xmlns="urn:xmpp:sid:0" id="e682bdd7-d98c-4cfd-9c59-fb9e5f9a6d8a" /><replace xmlns="urn:xmpp:message-correct:0" id="00dc00d3-ae5f-4572-b6c3-4b9e95445e5b" /><body>Shouldn&apos;t show up. 😂 </body><stanza-id xmlns="urn:xmpp:sid:0" by="xsf@muc.xmpp.org" id="2019-09-10-f3fa92f3f7cb7366" /></message>

heh

it's not my fault this time!

Daniel: no, just interested. I'd be glad to co-author as well

noo, poezio has only 2k lines in the xml_tab.. gonna grep logs now

didn't see that one in either place pep.

moparisthebest, neither did I, just looking at xml logs

pep., what's that?

moparisthebest: sigh

<message to="georg@yax.im/poezio-IS8H" id="718d40df-3948-4798-a99b-35cc9f03cc4f-13F5" type="groupchat" from="xsf@muc.xmpp.org/balu_der_baer"> <body>Can you see me?</body> <received xmlns="urn:xmpp:carbons:2"> <forwarded xmlns="urn:xmpp:forward:0"> <message xmlns="jabber:client" to="xsf@muc.xmpp.org" type="groupchat" from="xsf@muc.xmpp.org/balu_der_baer" /> </forwarded> </received>

It was a message that also contained a carbon

but that's ok to show up?

probably?

strange that dino doesn't show that one but Conversations does

i mean both is fine i guess

Yes, that's okay

so dino does have filtering? it's just wrong

What was that XEP that says "don't send everything in the same payload"

no i wouldn’t blame dino for not showing that

balu_der_baer: next time add another body to the carbon

moparisthebest, no it just goes down the carbons pipe

and then the carbon doesn’t have anything

Ge0rG, well that will show up in dino

but with the message from within the carbon

out of curiosity, can you put carbon into carbon?

no

pep.: https://xmpp.org/extensions/xep-0226.html

right that

i mean, what will clients do if they receive carbon within carbon?

just ignore it

I'm just awaiting the circular fastening

or parse the outer body if there is one

Daniel, that's what you'd hope they do

well at least for dino (even with the bug) and Conversations

and almost def Gajim

until we bring full stanza in the mix

then other funny things might happen

U+061C: only if the client has a recursive carbon parser

it'd be odd to have code to parse carbons recursively

yes

any clients written in lisp around? :D

that emacs client?

moparisthebest: a message parsing function that extracts the forwarded payload and passes it to the message parsing function? Sounds rather plausible

but hey with Xabber doing their own thing we will soon have new CVE instead of having to recycle the old ones

could be

Zash, unrelated, what conversejs version is running on xmpp.org btw?

Probably just the CDN version.

ah it says on the page

5.0.1

Anyone knows if this is the latest version of Prosody running here?

It's not

/version xmpp.org

is today the picking low hanging fruit day?

s/day/week/?

i also kinda want to rewatch BSG now

All this has happened before, and it will happen again, and again, and again

so yeah since people have started to exploit the dino roster push i should probably take this offline

Anyone got examples of strings that'd be different between IDNA 2003 and 2008?

Ha