XSF Discussion - 2020-09-20

Also something about backward compatibility and changing unicode versions

but is precis not better equiped to deal with changing unicode versions?

i read precis defines what code points are valid, while stringprep defines what code points are illegal

so whats legal changes with each unicode version in stringprep

lovetox, no, stringprep is pinned to unicode 3.2, so that’s no big deal

ah ok

didnt know that

lovetox, PRECIS on the other hand is not pinned to any unicode version, has no logic to deal with differing versions and thus PRECIS on Unicode X.Y may easily produce/validate strings which are not allowed with PRECIS Unicode X'.Y' for X' != X

stringprep being pinned to 3.2 is why we cannot have robotface ;)

sure? when precis defines whats legal instead of whats illegal

it does not matter what unicode version

it defines it in terms of unicode categories

if we assume that new unicode versions only add codepoints

yes

if you run PRECIS on 3.2, it will reject strings PRECIS on 9.0 allows

and updating unicode versions in an app is even harder than updating TLS, I’m afraid

could be as easy as "apt-get install unicode-data"

as i understand the problem is not some unicode library, or a dependency ✏

its simply that in the standard sometimes things change

the standard changes because with newer unicode versions codepoints that where previously unassigned become assigned

yes, but thats not the only way it changes, and thats not the problem with precis

I am not aware of other ways

Changes to the properties of Unicode code points can occur as the Unicode Standard is modified from time to time. For example, three code points underwent changes in their GeneralCategory between Unicode 5.2 (current at the time IDNA2008 was originally published) and Unicode 6.0, as described in [RFC6452]

new unicode versions are not just adding stuff on top, sometimes existing stuff changes

or at least there is no guarante that existing stuff does not change by the unicode consortium or whoever decides the stuff

true, and that could mean that a string that was previously valid as JID part becomes invalid

I am not sure how frequent that is

What usually happens is that a string that was previously invalid as JID part becomes valid

yes, also im not sure if this is really something that should hold us back

we are not designing something for eternety here

I'd expect that Unicode tries to prevent re-assigning codepoints whenever possible, for obvious reasons, and only does so if it is decided that the advantages of the re-assignment outweigh the disadvantages ✏

also i think adding new codepoints is not a problem for precis, as its already defined on classes

and there will likely be no new classes

what is a problem are for example mapping rules

> Future version of Prosody won't allow 👁🗨 or robot face in nicknames, thus solving that problem. Seems it's not yet in trunk.

which can’t be set in stone without knowing all future codepoints

✏

right, but it means that apps should use the system unicode database, e.g. via pythons unicodedata library that IIRC uses /usr/share/unicode as source

Ah, it worked.

i think its very unlikely with precis that something that was valid becomes invalid

hence i dont see why not use precis

yes servers and appilication need a recent unicode version

yeah, it also does not concern me much, and there is nothing you can do about it

but they need also X other recent librarys

mdosch: It's the MUC that enforces it, and this channel isn't on trunk.

The real problem isn't which version to check against but who's responsible for the check at which border. If the check is only performed by the server owning any given JID, no problems will arise.

If you enforce precis on another server's JIDs, you'll end up kicking people because of mdosch👁🗨 in your MUC

Hence being strict on creation of users, chat rooms, MUC nicknames.

We could solve the problem by requiring a baseline set that's forward compatible, like no " in JIDs, and leave everything extended unicode to the authoritative server

But I want 🤖

AFAIK the problem is mostly about unassigned codepoints, where you don't know if it's valid or forbidden.

So someone on Unicode 3.2 doesn't know whether 🤖 is valid or not, so it'll end up allowing it in JIDs received by others, while forbidding local things from using it.

Zash: allowed from remote servers, forbidden on yours

Exactly

And then your admin can install unicode 11

And allow fancy new names

I wonder if Someone™ should write an Informational XEP on this whole mess.

Zash: you should

why do we need to write a XEP

either all should use precis or all should use stringprep

That's not what reality looks like

finished, the details that there is one codepoint in a million that from unicode version X to Y changed

really, thats only a problem in some people minds

I'm not that worried about Unicode redefining characters between versions.

But versions add new characters, which moves code points from Undefined to either Allowed or Forbidden (for each JID part)

Zash i dont think thats how it works

Ok.

I revert to my earlier statement of not wanting to discuss this.

but even if, whats the problem with that

say an unassigned codepoint is moved to valid

your server simply does not accept it because you are on a older unicode version

there is no problem there

its like jabber.org does not allow connection with another server because it is weird and runs not current software or is misconfigured

the solution is not, to find a standard where this can never ever happen

its server need to upgrade from time to time

and it happens that we do this already

Oh look, it's been a year https://mailarchive.ietf.org/arch/msg/xmpp/a-WhzOTyOq168GujQHgzQ1-DURI/

yeah i really dont know what the problem here is

its like, a server comes along that supports only TLS 1.3, but the other server does not yet support TLS 1.3

and the question answer is probably b)

server should validate jids

this means sending errors if the validation fails

That's what jabber.org does, which was the problem highlighted earlier.

^ happens

For those who don't have joins & parts shown: ---> jabberdotorguser joined the room <--- jabberdotorguser has left the room due to an error (Kicked: jid malformed)

lovetox: enforcing validation on entities outside of the user's control is going to cause pain. This is what it's all about

Which is why "allowed" and "forbidden" are too few decision choices

about what pain are you talking?

informing the user he cant join this channel because the jid is not valid

is not pain in my book

its a 5 second thing to change the nick to something else

They can't change *someone elses nickname*

?! the user cant change his nickname?!

and yes also the MUC can change his nick, its in 0045

simply remove the offending chars

You still misunderstand.

This is not about the one that is joining a MUC

This is about someone else that is already a participant.

When the MUC sends the participant list, their server rejects that stanza and the MUC responds to that error by kicking YOU.

yeah and ? as a client a validate JIDs, and of course simply drop all invalid

you can fill a whole MUC with invalid participants, not a problem in my book

but even that should not happen

But it does.

So i cant connect to servers, if mine is outdated and uses old unicode data

Get a jabber.org account, join this MUC, get kicked the instant the presence of the participant with "👁🗨" in their nick is sent.

its the same, right now, i cant even connect to must mucs because my cert is expired

lovetox: but you can't change a remote server and which level of unicode that accepts.

Also the unicode level supported by a server is neither indicated nor negotiated

Instead your connection gets terminated later on due to somebody else sending presence

And just moving on with the latest and greatest unicode will break your interop

In all sorts of non obvious ways

it's like showing an annoying popup every time you receive something from an "invalid" JID :D

> Get a jabber.org account, join this MUC, get kicked the instant the presence of the participant with "👁🗨" in their nick is sent. What a joy.

Ge0rG, i still dont see the "pain", all that stuff is dependent on how often this happens

and i would say it does probably happens as often as you want to use a muc on a server like jabber.org

you try it, ok server doesnt work, is outdated, whatever, then you simply dont use it anymore

lovetox: some implementations don't switch from stringprep to precis because of this sort of issues that it would cause.

yeah, let's just abandon large parts of our ecosystem

but stringprep causes the same issues

larma: how do you prvent everyone from using the same nickname with 0172?

stringprep is obsoleted, no new client would implement it, there is no note that says: Hold up please implement stringprep

if a client uses precis, and the server validates for stringprep

you have the same issue already, now

Ge0rG, either not allow it server side (filter stanzas that do try to mimic another user) or use 0421 to spot the different users

lovetox: yes, and I bet most clients won't even tell the user what the problem is

yeay, just like my problem persists with trashserver <-> jabber.fr

so you acknowledge that the problem is already here right now, and *not* changing to precis

does not make anything better

lovetox: the problem is there because some implementations changed to precis, yes.

lovetox: what you ask for is called a "forklift upgrade" and is not going to work.

it already works

users use precis day in and out

this is a drop in the bucket of s2s problems out there

you make it seem like the whole xmpp ecosystem breaks down, because people cant join mucs anymore

it's also about contacts with JIDs according to a different spec

IMO clients should never try to join using a unicode resource, but servers still need to handle it. Yet every client that allows to do it should be named as the main issue

(which means about every client nowadays is to blame)

ok larma interesting take, no client should allow a user to use a valid JID as per RFC.

the weird thing is that resource is meant to be something "technical" yet it's also used as a display name

i guess you wont win that one

larma: what about that clients should only warn the user when they try to set a nickname that is outside of the client's supported PRECIS, but the servers have ultimate authority?

congratulations for finding out that MUC is a set of dirty hacks.

hi!

so let's add some other dirty hack to solve that problem? Clients somehow encode unicode chars using ascii as a resource and add a 0172 nick. Clients that see that the 0172 nick matches the ascii encoding will display and use the 0172 nick instead

As long as the ascii encoding is somewhat human readable, this would be sufficiently backwards compatible

then we would need MUCs to not allow joining with non-ascii resources and issue is mostly solved

Forbidding non ASCII is bad for Russians, Arabs, Vietnamese, Thai…

also for emoji

mdosch, it's not forbidden, you can still read it in the 0172 nick field

just like domains don't forbid non ascii, you just need to encode using punycode

what about encoding punycode nicknames in the resource?

also fine with me, but I believe there could be better legacy fallbacks than punycode

like... PRECIS?

Went for a walk. TL;DR let's solve the problem of not everyone upgrading at the exact same time with "just upgrade at the same time"? :)

its not a matter of upgrading at a certain point in my opinion

right now prosody does not do jid validation at all or?

that means it already is a upgraded precis like server, it does exactly what Ge0rG fears, it sends precis muc resources to other servers that dont understand it

It does, but it allows unassigned characters.

Mostly because this is the library default

yeah, so nobody cared, the ecosystem did not break down

No PRECIS

Somebody needs to care about the small things as well

you allowed resources that were not stringprep valid

and send them to other servers and clients

This goes under "historical reasons" now

The plan is to change it to be strict about things created locally.

and in all the years i never saw one issue, of clients or server operators

yeah, let's just pin XMPP to stringprep and carve that in stone

that complained that users cant join your MUCs

Zash: strict according to what spec?

This /did/ happen, for years, if you compiled your Prosody differently

🅶🅴0🆁🅶: Ancient STRINGPREP

Zash: did I get this right, you want to make prosody strict according to stringprep?

Yes

Insanity!

No

But why?

I'm running a version with this enabled and I'm having no problems.

You just can't use 🤖 as nickname on my local MUC instance

do you even have a MUC domain?

It's strict about *local entities*

I want my robot face back!

Local users, local MUC JIDs, local MUC participant nicknames.

Anything coming from a remote server and isn't known to be invalid is accepted.

Zash: yes, but then you release it and everybody goes back into 2002

I know there are people who wish for xmpp to be like it was in 2002.

ITYM 2006*

seems ejabberd does validate strictly for stringprep

cant join any muc with robotface

Zash: https://tools.ietf.org/html/rfc3454 "December 2002"

although it does with a weird error

🤷

bad-request

instead of jid-malformed

maybe the XMPP needs its own Precis profile

that is in some way a better upgrade path

for mucs the discovery problem is easily solved

just add a feature into disco info

This topic causes me endless pain, I'll be under my desk, crying, for the rest of the weekend.

only in our heads, i dont think there are actual users having problems with that

i guess there is not even someone out there that gets the idea that weird emojis are allowed in jids

I do

and other people as well

Ask Rixon 👁🗨:

as you can see on the occupant list. Unless your client filters out "invalid" JIDs from MUCs

I wonder why Rixon 👁🗨 never participated in this discussion although being highlighted frequently in the last days.

mdosch: maybe their client fails to highlight on complex unicode? 😁

😂

I will do a little test for MattJ

^ connection and disconnection in less 1s

Hey anyone here

heyy

hey sony

andrey.g andrey.g \