XSF Discussion - 2020-02-14

Yea I saw that too, though previously I heard it was dead in the water due to middleboxes, so I wonder if anything changed there

moparisthebest, what if you want to resume a connection initially established via quic on a connection using websockets as transport binding? ✏

jonas’, "via stream feature + attribute on <enable/> as negotiation" →

why the attribute? what's its value?

flow, the client needs to tell the server that it supports alternative transports, too

it could be "supported-transports='...'" or "i-support-quic='true'"

jonas’, why does the client need to tell the server that it supports alternative transports too?

flow, so that the server can pick a proper reconnect address out of multiple options

jonas’, ok, but I still think we can s/need/should/

we don't have enough handshakes yet.

Ge0rG, I don't think there is an additional handshake involved

flow, if you say so. I also think that there’s little point in discussing the details right now

Not sure which time would be better to discuss matters of xep198, especially since there is a CFE for xep198 out

it’s not about '198

it’s about future extensions

maybe you should move this to the ML

it's CFE after all

Is this experience? Seems like speculation about future things

Do we have a formal Call for Speculation? I'd love that.

Experiential part is that people do use '198 resumption across different transports.

well the question is if those future extensions require adjustments to xep198, so I think it touches at least xep198 and its CFE, especially since its the step before xep198 becomes final

Ge0rG, I think I posted everything I said here also to the ML. Please correct me if I'm wrong

flow, my rough draft of how such an extension could negotiate features was meant to show that it’s possible *without* having to modify '198

the details on whether a step of that is necessary do not matter; it is meant to show that a new feature can be introduced in such a way that both parties are sure about whether the other side supports it.

Can anyone think of anything that would break if a server was going to Unicode NFC-ify everything that goes through it?

Kev, xep426

for certain values of "would break"

Ta.

Kev, Everything? Yeah, anything relying on character counting. You're NFC rather than NFKC, correct?

NFC is what I'm currently having a conversation about.

NFKC is the lossy one, I think?

NFC is more stable between unicode versions, so should be OK. Anything that relies on digital signatures will/might break too.

NFKC is Kompatibility, so ™ -> TM ✏

But mostly, I'd have to ask why you'd want to. NFC is quite expensive, after all, I wouldn't want to normalize everything if I didn't have to.

Covert Channels.

Side channel attacks based on composition? I suspect that word choice would be far simpler, but I suppose it's worth closing.

Yes, it feels to me that it's closing a window when the door's open.

With the dozens of different whitespace characters, you can't have that anyway.

Whitespace is normalised in canonical XML isn't it?

Kev, Oh, if you're canonicalising XML too, that might be fun. And no, whitespace is not canonicalized, except outside the document.

Kev, Excepting line endings.

<this is= 'though'/>

Depending how you define 'outside the document'.

Kev, Yes, inside start and end elements it's normalised. But in any text it isn't.

Right.

Kev, You'd also have to deal with people randomly slipping in additional attrbutes.

Quite.

dwd, "in any text"? with xml:space="preserve"?

pep., I think that's the case by default, isn't it? I see, though, that whitespace is normalized in attribute values.

I’d very much like not to set a precedent for a server messing with data

All whitespace in attributes seems to turn into 0x20

jonas’, That precedent is already set to some degree - lots of cases of linking high security networks with lower security networks where data is heavily futzed about. But I agree I wouldn't want to see this normalization behaviour normalized.

indeed

dwd, I'm curious if XMPP defines anything re whitespace actually. I'm reading §2.10 White Space Handling in XML 1.0 and it says "In valid documents, this attribute, like any other, MUST be declared if it is used."

https://www.w3.org/TR/REC-xml/#sec-white-space

pep., i'd assume that xmpp operates with xml:space='default'

Is it defined what default means

somewhere in XMPP

pep., ha, right XML says it is then applcation defined and we don't do that

at least to my knowledge

so anything goes?

¯\_(ツ)_/¯

I wouldn't be surprised if most XMPP implementations preserve whitespace

besides some doing a trim() on <body/> (dino IIRC)

I think they do, I sometimes see pretty-printed XML in my client logs

flow, yes we do trim() on text elements, but that was only because some implementation would pretty print by adding newlines at the beginning/end of text elements...

fix the sending entities rather than trimming? :/

larma, i have a dino user which reports that dino does not show whitespace only messages. Is that correct?

probably

which lead to the funny situation that the whitespace sending person would send an "sorry for that" message afterwards, which confused the dino user because he didn't knew for what his contact was sorry for ;)

not saying that dino's behavior is wrong, just telling the story…

pep., well it's funny that pretty printed XML is different to non pretty printed XML in XMPP.

larma, different how?

<body>Test Test2</body> usually is pretty printed to <body> Test Test2 </body>

ahh, that implementation would add newlines within the textual content of body

larma, which implementation was/is that?

larma, ask the sending entity to fix it rather :/

It's their fault not yours. And trimming on reception is only going to harm your users

i'd definetly describe that as broken, as one can argue that XMPPs xml:space="default" is preserve

pep., I am not sure how much harm it causes (besides the story I told)

Also some XML implementations also (at least have an option to) a) consider all whitespaces the same and b) merge multiple consecutive whitespaces in a single one

I find both a) and b) in XMPP as crap as the trimming you describe now :x

larma, i am still missing the part where we publicly shame that sending implementation ;)

I'd say the behaviour is always preserve in text nodes

pep., both a) and b) would work everywhere in XMPP except for the body.

larma, or anywhere that defines containing text

pep., well I don't know too many examples but I remember that it worked for some

xhtml-im. possibly reactions? (when are we going to be able to send more than one grapheme cluster? :p)

I *hope* XHTML-IM does the same whitespace handling as XHTML, which is the one described above IIRC

But if the XML library used in the XMPP client doesn't..

Yeah, funny things happen there.

I am not aware of any HTML renderer printing " " as two spaces except of inside <pre> or <code> elements

XHTML defines xml:space="default" as both your a) and b) right? and code and pre are xml:space="preserve" (or only pre?)

probably

Also it's funny that whitespace handling can be changed at the CSS layer

unrelatedly

If mixed content is illegal in XMPP and whitespace only textnodes should not be ignored, don't we get illegal XMPP XML as soon as you pretty-print?

I guess we do :x

maybe it's good that we haven't clarified whitespace handling too much in XMPP

somebody(tm) should open an errata I guess

can we maybe go with xml:space="preserve" on <body> and basically ignore whitespace outside?

Even if it's not what Dino does, it seems the most sane to me

no we can't. We should recommend XEP authors to take tht into account

and properly define what is xml:space="preserve" and what isn't

if it's not the default, that currently isn't defined anywhere

pep., what should they take into account? If they don't account for it default applied, so we just need to make default to ignore whitespaces pretty much

larma, why not simply assume preserve everywhere and say that sending pretty printed XML e.g. <message>\n<bar/></message> is invalid in XMPP because it is mixed content?

flow, yeah that's be a good first step I think

flow, you want to break backwards compatibility?

also, why don't we like pretty printed?

maybe even with 'default', <message>\n<bar/></message> would become <message>U+0020<bar/></message> and still be invalid

larma, I don't think I am breaking backwards compatibility

flow, no, "default" not being defined in XMPP there's no normalization happening

(besides that, I usually try to avoid breaking backwards compatibility)

I am used to typing XML directly in my openssl s_client and it helps a lot that I can put newlines arbitrarly except for body

larma, it's already the case no? That mixed text/tag is not valid

I see, but the client could also "normalize" your pretty printed input before its put on the wire

pep., yes and no: text might be but whitespaces are ignored

Do you know where that's defined?

flow, so how do I put malformed XML on the wire then? 😀

pep., maybe nowhere

larma, hehe, I'am sure you figure out a way

pep., I just tried: `<message to="jabber@larma.de">Bla<body>Bla2</body></message>` was forwarded by prosody and rendered as "Bla2" on Conversations and Dino

:(

This message is invalid.

Did anyone *not* see "This message is invalid." ? at least nobody seemed to have send an error and the server forwarded it as is (just added a stanza-id)

poezio displayed it

I am not surprised that the "no mixed content" rule is not enforced

flow, well if it's not enforced, enforcing it is likely to break compatibility...

most implementations will simply ignore the textual content

larma, I think I have a different definition of compatibility than you

I just checked RFC, and mixed content is allowed

It's just not allowed inside <show> <status> <priority> <body> <subject> and <thread>

ahh

6120 §11.7 "Except where explicitly disallowed (e.g., during TLS negotiation and SASL negotiation), either entity MAY send whitespace as separators between XML stanzas or between any other first-level elements sent over the stream. One common use for sending such whitespace is explained under Section 4.4."

0071 version 0.19 changelog: "Per list discussion, removed recommendation to preserve whitespace in XHTML bodies (instead, use of <br/> and non-breaking spaces is recommended)"

whitespace handling in XMPP is :x

#FightForWhitespaceRights

I think the RFC was not considering details of whitespace handling in body. A client that replaces new lines with spaces in body would probably be RFC compliant.

All examples in RFC are pretty printed but no note that those string won't actually be valid. https://www.rfc-editor.org/rfc/rfc6121.html#page-74 is an example where pretty printing adds new lines at begin and end of text content

So the client that did cause Dino to add this trimming was actually RFC compliant

and it is just not properly possible to send a message that starts with a new line

Why not?

This has 3 newlines at the beginning

because the newline could be added as part of pretty printing

which seems to be fine according to RFC

one newlines here before, and one after

#FightForWhitespaceRights

This message has some spaces

(funnily enough Dino does not do trim before sending, so my message is displayed differently on sender and receiver)

Conversations also does the same trimming as Dino

https://ppjet.bouah.net/whitespace.png

This message is pretty printed RFC style and thus should probably be displayed without leading or trailing newlines or whitespaces.

Hmm

Ha, conversations really trimmed the newlines ✏

So status is: - Servers do transfer XML verbatim including unnecessary whitespaces between nodes. They just add elements before the closing </message> if necessary. - Some clients ignore leading and trailing whitespace of all kinds in body (aka trim()) when others don't

ejabberd won't include whitespace between nodes FWIW. But I ignored the backlog, probably not relevant :-)

s/include/preserve/

Holger, between nodes or between stanzas?

I'm actually not sure what Openfire does.

larma: Between XML nodes.

I'm firmly on team #trimWhiteSpace in the presentation layer though. But we've discussed that to death already.

larma: All incoming traffic is decoded into internal structs and it won't include unnecessary whitespace when encoded for transmission.

Holger, but not inside XML nodes? <body> </body> will go verbatim whereas <test> <test2 /></test> will not?

"unnecessary whitespace" I don't think that's defined in the RFC 😀

I don't expect trimming text in text-nodes is allowed, is it?

and if I recall correctly, XMPP does not allow for mixed nodes (what's the correct name again?) where you mix text and elements in one node.

Guus, we just checked, it is allowed in general just not inside certain text only elements like <body>

larma: Well I just had the `<test/> <test2/>` case in mind, which you probably don't care about. In your example "it depends".

larma: you're only talking about whitespace though, correct? I ment: <outer>some text<inner/></outer>

Guus, no I was talking about everything

I was under the impression that something like <outer>some text<inner/></outer> is invalid in XMPP - but I might be wrong.

There is no general rule if RFC 6120 and RFC 6121 has specific rules to forbid mixed content inside certain elements

Also mixed content is allowed in XHTML-IM which is in XMPP 😉

I think the consensus is that we don't like mixed content, but it's not disallowed by RFC

meh. I'm looking forward to your bugreports. 😃

As I see some people don't care about whitespace and treat it horribly (#FightForWhitespaceRights !), would you respect xml:space="preserve" btw? If I say on body "don't touch plz", would you stop trimming etc.? ✏

Or because it's the assumed default would you continue not caring

pep., you mean, you'd like to have <body xml:space="preserve">\nTest</body> have an empty line displayed before the message? What is the use case for this?

So that you respect my whitespace

Whatever usecase

I am respecting the whitespace, I am just not wasting my users screenspace

You're not respecting whitespace if you trim it

power users would probably appreciate that, if only them

To me this is about the same fight as "don't interpret body if I don't say so"

By now I honestly think that newlines were completely out of scope in the RFC i.e. it might be that it was not intended what we do nowadays with newlines and that you instead just send one message for each line.

pep., an empty line can just be invisible. This has nothing to do with me respecting whitespace or not, this might even be a font renderers decision (depending on what font renderer you use). Display and transport are generally independent from each other.

For example when I have a line with an emoji in it 👍️ this line will take more vertical space then a line without an emoji. consequently a line without any characters in it could be rendered without needing any vertical space

Also some fonts render consecutive spaces different to single spaces.

That's an issue with your font renderer then

It's not like I was asking you to display protocol-things verbatim to the user, like say: Hey somebody commented on your post: ""

I'Ve seen that in a client not so long ago for subscription requests when there's no comment

Just asking to display what's intended by the sending entity

preserving whitespace might be simple for terminal clients with monospace fonts (as if monospace fonts were still a thing), it is not for the general case

pep., this is strictly impossible because the sending entity does not know my client/font etc

So we might as well give up and close shop? :)

Or do all the weird things we're inflicting onto body?

Because it's not possible to do it right anyway

What are you expecting this to look like: a b

dunno, characters and a bunch of spaces

there is a tab character before the a, I might be expecting it to be displayed as 4 or 8 spaces or displayed as \t

If you use \t you're not expecting it to be displayed as any fixed number or chars no (programmers' myths)

(that's the whole point of \t)

at the beginning of a line I am

The sender doesn't decide of the number of chars that \t will be replaced with, is what I mean

Yeah sure, but I might expect you to display it like that

just because the information is not transferred doesn't say anything about user expectations

I get your point, but you've got wrong expectations here

We literally have a GitHub issue with someone complaining that the ASCII art send from another client doesn't render properly, so users have strange expectations

Everybody should use monospace \o/

Being able to send empty lines is probably the least relevant expectation

pep., monospace is a lie

Do you want a screenshot of my desktop just now? :p

WWW W👍️W WWW My monospace font doesn't monospace this

hmm

https://files.mdosch.de:5281/upload/zHk1fWzzbI53ZVxn/2020-02-14-160041_scrot.png

Haven't measured it, but seems quite monospace to me.

Those were three lines with each three characters in it

larma: I wouldn't expect emoji to be monospace

!XSF_Martin, so how many spaces would you expect it to be?

It's little images and making it the size of a character would render them toooooo small.

So they shall be arbitrary length? Isn't that completely contradictory to the purpose of monospace fonts?

larma: I, as a user, would not expect monospace to work with emoji. I'd rather be annoyed if you display emoji as small as one character.

larma: Although emoji are included in fonts I don't see them as *text*

I expect this to be same width, but not if I embedd emoji, images, stickers or whatelse the kids do today. ASD FGH JKL öäü asd fgh jkl

larma, would you also trim nbsp btw?

Maybe I should start replacing all spaces in front and at the end of lines by nbsp :)

!XSF_Martin, How about — this is the em dash, which by definition should be 1em width

pep., we trim whitespaces

Any?

nbsp under typical definition is a whitespace

meh

I haven't tried all of them

There is a line with full-width spaces above.

Yeah, nothing visible on any of me devices

larma: What about it? Seems to fit in well here. ——— ÄÖÜ @»« ł€¶ ŧ←↓ →ø[ ]}\

pep.: In profanity there's an empty line in your last message.

https://nl.movim.eu/?node/pubsub.movim.eu/Movim/87633da7-3963-4923-aabc-54ac5f6ad1d8

!XSF_Martin, yes

!XSF_Martin, well it if has the same width as any other characters and your don't use a square font, you are rendering the em dash *wrong*. em dash is defined that it should be rendered with the width of 1em. 1em is the same length as the font's height

edhelas, nice :)

We were talking about expectations, and I certainly expect a receiving client to follow the unicode rules for rendering the characters I send (or not support the character at all)

> !XSF_Martin, well it if has the same width as any other characters and your don't use a square font, you are rendering the em dash *wrong*. em dash is defined that it should be rendered with the width of 1em. 1em is the same length as the font's height No idea if terminus is square, but looks like it's not. Height should be bigger than width.

That's what most monospace fonts do, that's why I came up with this example 😉

Yeah, if you want to have your characters the same width you probably don't want the dash to stand out.

well I didn't ask for you to render characers the same width, I asked you to render that specific character with 1em width, I don't care what width you are using for all other characters.

Are you arguing that recipient's preference should override sender's preference?

Yes, I also wouldn't want to see red text on orange background only because the sender likes this theme on his client.

That's why I choose a specific font and color scheme.

It's the same reason I don't like people sending me HTML email without plaintext

But for the record: I am only speaking as a user, I'm neither an XMPP developer nor affiliated with the XSF

OK, so we agree it is perfectly fine that a client ignores incoming leading whitespaces, as this is a recipient's preference

So you'd also remove "a"s in a message as this is a recipient's preference?

if* this is

larma: I talk about *styling* like character width, colors and so on. Not about removing content.

A newline is not content

Same applies to whitespace

It is formatting

larma, the day I send whitespace (language), it is content :)

Hmm, I see it is hard where to draw the line.

larma, we both agree it's not a common use-case for dino's target

Your client is rendering many whitespaces the same (because there are various whitespaces that define a length that will become the same when displayed monospace), so you should expect my client to change rendering of whitespaces as well

But as pep. says, if you paste code snippets removing whitespaces might be bad. In case of common languages it might be cosmetics but in case of whitespace (language) it's removing your code.

I have hard times imagining any valid use case for leading newlines

!XSF_Martin: we are only talking about leading/trailing whitespace

And I don't really consider whitespace (language) a valid use case...

In this case I would be fine with it. But if you want to compile a general rule I'd go for "don't touch it" than "do whatever you want to"

What is "it" that you don't want to touch? The bytes on the wire or how they are displayed?

The problem is: the information about how to display things is lost on the wire, so clients have to do assumptions on how to display things. ✏

larma, FWIW, I use leading newlines when I paste a block of lines so that they’re all indented the same

many clients indent the first line more than subsequent lines (because the name is prefixed)

Isn't that a perfect example for using monospace format blocks?

Which start with a line containing ```

but you don't want to actually display the ``` except when it's not a block delimiter, which you can't know because there is no meta-data

Ge0rG: we can do the same with message markup xep if you don't want to display ``` ;)

or, y’know, XHTML-IM

minus the CSS crap

everything is horrible

Also dino does not indent the first line so this is a good example where it is again perfectly sensible for dino to strip the empty line ;)

poezio also doesn't indent the first line

it indents all lines.

but the /code plugin fails on pasting tab-indented code because initial whitespace is stripped ✏

everything is horrible

pep., so you are wasting a line of screen space whenever jonas’ sends a block of lines. And you are complaining about Dino wasting too much screen space 😉

Even if jonas’ were abusing it we would be wasting less space than dino :p ✏

was*

english?.

pep., "were", it's a subjunctive.

larma, current terminal size: 273x74

dwd, thanks

pep., But only foreign speakers would get it right. :-)

haha

pep., I suspect it properly takes an infinitive, as well, so "even if jonas’ were to abuse it".

right, that sounds better

anyone have a direct way to contact the Monal dev re: https://monal.im/blog/monal-4-3-is-coming-out-in-about-a-week-even-in-france/ ?

long story short he's silently dropping all OMEMO compatibility with all other clients because he thinks he has to due to a crypto library constraint, when he can likely keep support with a bit of code, I've pinged him with an offer to help in the official muc yesterday but no response so far...

moparisthebest, thats a bit misleading

or not, he is breaking compat with all clients, but most clients will update their code soon to send 12 byte IV

so that leaves older versions probably in stable distros

right, so breaking compatibility with most existing clients for the next few years until all users update

I suspect it'd be better for everyone involved if he simply kept compat with 16 byte IVs in the short term no?

yes correct

just requires a small algorithm to hash 16 bytes down to 12

he said its not possible with apples GCM impl

dont know if thats true

yeah is monal not open source?

you could just provide a MR

and while you are at it a problem is also chatsecure, which only accepts 16 byte IV

well I volunteered to help but can't write Objective-C or compile/test/run iOS apps

I can write what they need in C or Rust or something though

true

much overhead

but i guess the offical muc is the best place to reach him

yep to put in a MR I'd only need a macos machine for development, an iphone for testing, and $99/year for the privelege :P