XSF Discussion - 2020-09-14

Sigh

I think for XEP-0335 I received conflicting feedback :)

One request was to drop all mention of character encoding and anything below the "stream of XML characters" layer, and the other was a request to add details of characters that need to be XML-escaped

the former seems sensible

converting an arbitrary unicode string into its xml representation is not something you should cover in an arbitrary xep

it's something the the xmpp-core rfc covers by saying "we put XML encoded in UTF-8" on the wire

What about arbitrary Unicode? JSON is weird and requires that to be escaped to \uXXXX IIRC

it would be different if we talk about codepoints that re not allowed in xml 1.0

and XML forbids some characters (even escaped) that JSON allows

so I don't think it's as simple as "don't mention it"

And probably in UTF-16 surrogate pair mode

Zash, for those the escaping happens on the json layer, before the xmpp library sees it and transformes the unicode string to xml, no?

Sure, yeah

MattJ, are those characters kept in their naturual representation in json or, in the escaped representation?

MattJ: hm?

if the latter, then we are fine, if the former, then you have to either define your custom escaping scheme, or convert the json string to base64

Bring out the Venn diagrams!

https://mail.jabber.org/pipermail/standards/2019-February/035796.html

Zash, how does the choosen unicode encoding (UTF-8, UTF-16) matter?

Does it say somewhere that an encoded JSON thing is ASCII?

isn't json, just like xml, at first only a sequence of unicode code points?

Aren't JS (and hence JSON) strings by definition UTF-16?

but then they are also a sequence of unicode codepoints

But the encoded JSON is either clean ASCII or ???

I guess the encoded JSON is whatever encoding you agreed on to exchange the sequence of codepoints ✏

> JSON syntax describes a sequence of Unicode code points. http://www.ecma-international.org/publications/files/ECMA-ST/ECMA-404.pdf

it appears to me that json strings must contain e.g. control characters in their escaped form only

All Unicode characters may be placed within the quotation marks, except for the characters that must be escaped: quotation mark, reverse solidus, and the control characters (U+0000 through U+001F).

RFC 7159 § 7

so if you feed a json string to an xml library, the xml library should never see e.g. U+0010, but only the sequence of codepoints that composes the escape sequence of U+0010

Seems so

But waqas refers to https://tools.ietf.org/html/rfc7159

Why have one definition of JSON when there can be three?

The XEP refers to https://tools.ietf.org/html/rfc4627 which was obsoleted by 7159, I guess I'll update that

Or .. twentyeleven.. probably about as many as there are JSON implementations

https://tools.ietf.org/html/rfc7159#section-7 > unescaped = %x20-21 / %x23-5B / %x5D-10FFFF

Huh, where did I see the thing about surrogate pairs then?

If someone wants to tell me what characters are unrepresentable in XML, please add to the list thread I just revived and I can include it in the next revision

there are characters that can't be represented in XML?

According to Waqas. I don't see them at a glance

JSON characters, that is

Some control characters are unrepresentable in XML 1.0

It looks to me like they are also forbidden in JSON

Seems like they have a Wikipedia article: https://en.wikipedia.org/wiki/Valid_characters_in_XML

MattJ, forbidden as in, they have to be escaped in JSON, right?

Yes

or

I guess you have to check if any of those characters

unescaped = %x20-21 / %x23-5B / %x5D-10FFFF

is invalid/forbidden in XML 1.0

Yes, I don't see anything forbidding escaped things in JSON

Any character may be escaped.

says RFC 7159 (the current JSON RFC it appears) ✏

so a naive but safe json to xml converter simply escapes every character ;)

this errata, although rejected, may be of relevance: https://www.rfc-editor.org/errata/eid3984

MattJ, note that XML forbids control characters, even in escaped form

not that it matters for the JSON usecase

(here, "escaped form" means hex-entity-encoding)

jonas’: I know, that's why I chose "unrepresentable"

FYI, we just received our GSoC payment. Thanks to everyone who contributed to this year's summer of code!