XSF Discussion - 2019-07-27

It seems I fixed everything.

Now OMEMO works all right in my client!

“20:49:04 Zash> Actually, you can probably add tags in a custom namespace if you want”, clients will parse the namespaces into their internal structures, then serialise them back without including your extensions, sorry.

If you want do extend your bookmarks, you’ll have to use another private PEP node.

unless bookmarks 2 or something specifies that stuff needs to be kept :)

jonas’, as far as current implementations are concerned, Bookmarks 2 doesn’t exist.

jonas’: keeping unknown elements in a structure that you have to parse is a nasty cross layer challenge.

yeah and i would rather not rely on other clients not overwritting my custom elements

Using a private pep node for that, you dont need to change a xep, you dont need to rely on other clients, you can implement it right now, and it will just work

Where do I send patches for the website?

editor@xmpp.org or somesuch?

Zash, that should be fine, or you could use GitHub and send a pull request.

Editor will be the one merging it in the end.

I don't feel like interacting with github today, and I strongly believe that it should not be required.

Especially now that it’s impossible for people living in countries the USA considers as enemies.

Well the XSF is incorporated in the US so that point might be moot :/

The XSF isn’t enforcing that ban on any of its infrastructures, AFAIK.

GitHub is, since a few days ago.

Maybe this merely reminded me to exercise the non-github patch submission path :)

:)

Zash: how do you obtain a copy of the repository without interacting with github?

Ge0rG: Magic already existing local clone.

https://xmpp.org/about/technology-overview.html#pubsub has a broken link to what I assume was a local render of https://tools.ietf.org/html/draft-saintandre-atompub-notify-07

What's the relation of that and XEP-0277 (or is it 227? I always mix those up)

hope we can use salut a toi one day to send patches via xmpp :)

Store your code in PubSub, not in GitHub!

I just got into an argument with someone arguing that XMPP shouldn’t be able to send/store files, even though that was just HTTP. :(

XMPP shouldn't be using HTTP for sending files.

Agreed. ✏

hm if set stream lang to german

i should not have to add this to presences if i join a muc or?

In theory

Which server?

prosody

Zash what does this mean if i set xml:lang=de, and prosody answers me with xml:lang=en

is the lang set on each direction of the stream separate?

I don't know. Prosody doesn't do anything with xml:lang

lovetox, yes, each direction is separate.

it does, because it sets it to en :D

Zash i hope it adds whatever xml:lang i set to whatever stanza it routes to other servers

Nope

It does *nothing* with xml:lang

...

But yes, the server SHOULD (IIRC) add the stream’s @xml:lang on each outgoing stanza if you haven’t set one already.

Except if it's set on a presence stanza that creates a MUC it uses that as default language in the MUC config

Is there a particular reason prosody ignores stream xml:lang attr?

Ah no, it’s not a SHOULD it’s a MAY.

“If the initiating entity included the 'xml:lang' attribute in its initial stream header, the receiving entity SHOULD remember that value as the default xml:lang for all stanzas sent by the initiating entity over the current stream. As described under Section 8.1.5, the initiating entity MAY include the 'xml:lang' attribute in any XML stanzas it sends over the stream.”

lovetox, it’s allowed in 6120.

Section 4.7.4.

no its a SHOULD for the server

Ah right, I misread.

Link Mauve, Wait inheritance is not mandated by XML itself?

Zash, stanzas are sent to other entities than your server, so that wouldn’t be enough.

Zash, im aware that prosody does not provide translations

but other servers do, and it would be nice if prosody adds whatever stream xml:lang i set, to outgoing stanzs

Nobody has written code that does that

I'm surprised that this is not a MUST in the standard.

Same.

I've just added xml:lang to yaxim yesterday.

What would make a MUST better here?

flow, clients wouldn’t have to copy it manually on each stanza.

I assume we are talking about the first SHOULD in the quote link posted

I wasn’t aware of this particular issue.

Link Mauve, well a SHOULD is already pretty strong. I usually read it as "you have to do it, unless all invovled parties aggreed on something else"

That way, you could write a simpler service implementation, use it in a, more or less closed, envrionment, where that requirement is not needed by every involved party and still claim standard compliance

> If an outbound stanza generated by a client does not possess an 'xml:lang' attribute, the client's server SHOULD add an 'xml:lang' attribute whose value is that specified for the client's output stream as defined under Section 4.7.4.

This is §8.1.5

flow, I actually read MUSTs that way, as long as both entities agree they can do otherwise.

Inheritance of xml:lang is a huge can of worms.

I know that endless discussion have already take place about that topic at various venues :)

Link Mauve, so what makes a SHOULD different from a MAY and from a MUST in your book?

I'm sure there were.

flow: prosody developers will implement MUST on request.

Ge0rG, SHOULD too.

Link Mauve: not in my experience.

flow, hmm, SHOULD is more like, it’s possible to not implement it that way if you can’t, for a good enough reason.

And MAY, do whatever you want.

ack on MAY

but I guess if we ask enough people we sure get two contradictory statements on that too ;)

Yeah. ^^'

It feels like your SHOULD interpretation is not so far away from mine

so in my book, MUST is something that you can not just not follow, as otherwise some very important aspect will fall apart

like "MUST be initialized with a CSPRNG"

Ge0rG, can you give an example why this is a can of worms?

of course you are free to ignore that in your implementation, but then you can't claim standard compliance and you have to live with the consequences

Earlier today I was reading the XML specification, and found not respected MUSTs in minidom which are totally fine not to respect in our XMPP environment.

lovetox, some XML parsers, IIRC the one on Android, will not inhert xml:lang to deeper XML levels

Incidentally, I just opened https://gitlab.com/xmpp-rs/minidom-rs/issues/17

lovetox: that, and your prosody issue.

that means you may have to work around that if you want get the effective xml:lang

note that this is usually also true for namespace prefix bindings

Also displaying of the right human readable text from a message with a set of different language texts

i have the feeling you talk about some use cases that never happen

also i dont see why i would need my parser to do some inherit stuff with xml lang

They never happen because nobody is making use of xml:lang because they never happen.

this is about telling my server my preferred lang, i dont parse it anywhere, i only send it

lovetox, well let's say there is a server which optimizes xml:lang by stripped redundant ones

I think there are some more cases, but yes it obviously hasn't hurt xmpp that much in the past 20 years

lovetox: the body inherits the lang from the message, which inherits it from the stream. If your xml library doesn't do that Inheritance, you need to manually traverse the hierarchy

on the other hand, smack just got support for an xml environment which keeps track of xml:lang

which is hopefully caused by xmpp becoming more and more used so that those rare cases where xml:lang inheritance becomes important also become important ;)

no you really dont Ge0rG, at least not for the basic use case of getting translated messages from your server

what you are talking about is user to user stuff

yeah i wouldnt even start with xml lang there

lovetox: I'm talking about the code needed to display the correct string to the user

lovetox: let's say it's not message body but the text field from a message error

Same situation

Ge0rG, aren't you talking about the "here is the same error message in 20 different language strings, please select the best one" case?

whats the problem? you dont work with inheritence there

you just set the xml:lang on the <text>

the server sets it in this case

if its not there, its the lang the server told you in stream init

flow: I'm talking about the smack report where no text at all is returned when the languages don't match

Ge0rG, i guess that is very indirect yes to my question

lovetox: you just described Inheritance

flow: maybe not 20, but 2, or even just one, but in the wrong language

why would a server return 2?

this is already some theoretical stuff

lovetox, because the stanza may not originated from the server

it maybe came from a remote entity which has no knowledge about your stream's xml:lang

lovetox, for instance, so you can have a nice localised message for user display, and the English version so they can report to the admin and easily look it up in the code.

yeah still dont see the problem, one has xml:lang=en the other one "de"

If I were to implement localised error messages in a server, I’d do it that way.

Link Mauve, +1

<error> can have "fi"

it does not matter

lovetox, the other one may not have it on the <text/> element, but on the stanza.

because the directly set xml:lang on the text will always overrule parent or not?

It still MUST be understood as German.

lovetox, yes it will.

But the @xml:lang might be present on any parent of the <text/> element, and you must inherit it in any child which doesn’t have one defined.

so basically if you have: stream xml:lang=en, stanza xml:lang=de, error xml:lang not set, your parser should see the error's xml:lang as 'de' and not 'en'

or, even better, your parser sould provide you mechanism to determine the effective xml:lang of the element

And not '' either (which means no language set, as per XML 1.0). ✏

Link Mauve, yep, today I sadly learned that the empty string is valid for xml:lang

TIL too!

another fun fact: xml 1.0 mentions BCP 47 for xml:lang, xml 1.1 mentions rfc 3066

now go out and find out the fun :)

does the rfc not define one them self

rfc says BCP 47

ahh you spoiled the fun for the others, yes rfc 3066 is bcp 47

:p

it's not as bad as stringprep vs precis, eh?

I just find it strange that this changed between xml 1.0 and 1.1, and i am curious to get the rationale behind that (if there is any)

flow, isn’t BCP47 a live standard, as in it will point to whichever RFC is currently considered the one?

either way, servers in general should get translation going, i see it working in ejabberd, its a nice thing

Link Mauve, are they? but yes, that could be a reason

lovetox: there should also be a translations project for the common error conditions.

translate your favorite error message!

This is especially important for stream errors, where the text element is an extension for the technical details, not the actual explanation

> lovetox: there should also be a translations project for the common error conditions. +1

yes also common forms

Those too

Yes please, Someone™ do that.

The JSF should do it.

Do we have a complete list of all places where the same translations are needed for all implementations?

Forms, error conditions,...?

Ge0rG, take a translated server’s .pot file?

forms and error conditions, i didnt come across any other places where a server adds userfacing text

lovetox, user configuration maybe?

Link Mauve: there is no 1:1 mapping between conditions and server side error messages

whats user configuration?

lovetox, some servers may expose e.g. a web interface to their admins.

Ge0rG, I know.

Create a repo, stick some .po files or whatever in it, pre-seed it with text from the RFC?

Zash: on github!

:D

i think one would need some translation website, like pootle or something

Not strictly, but such a thing can be used

Transiflex?

Launchpad?

transifex is pretty expensive

Are you supposed to do xml:lang inheritance yourself or can libexpat do it?

Zash: yes.

.

Why would you care as server?

just set the attr on the top element

and be done

lovetox: The question is "Can the XML parser library do the thing you just asked us to do?"

setting an attribute on a stanza?

Inheriting xml:lang from the stream to child elements

im pretty sure it can do it

I'm not sure you grasp just how deep this can of worms is

If something copies one of the child tags of the stanza, they should probably also carry the xml:lang

Ie it should work similar to xmlns, which the xml parser supposedly handles for us

I think you overthink that

all you have to do is, add the attr to the message, iq, presence, and leave the rest to client parsers who get the stanza

of course only if its not already there

and this can never be wrong, as we set it on the stream already, so it is inherited in message, iq, presence anyway

you just make it explicit

you dont change anything here

and thats only because these stanzas will reach entitys which dont have the c2s stream context

and nobody asks you should make it explicit all the way down to the last child

only the childs of stream

lovetox, think about some user sending the server an xml:lang="fr" message, the server then copies only the body into some other message, maybe to send it to the admin, or to archive, or anything.

The original body didn’t contain any @xml:lang, but the copy should contain it as xml:lang="fr".

storing and restoring messages and all of its content correctly is a general issue that does not stop at that attr

xmlns, xml:lang, something else?

yes if the server does not preserve attr on the message

there needs to be changes to make this right

lovetox, it’s not only on the message, it should be “propagated” to all children of the stanza in that case.

but i would never go the way and add the attr on all childs only because i dont want to store it on the message

No Link Mauve server should replicate the message exactly

if i send a message with lang=de, i expect it to be routed that way

and not with lang=de on all childs set

i mean yes it would be still correct

lang=de is implied

but weird to rewrite a stanza that way

and you do it with jabber:client already?

why dont you copy it into every body?

magic

It's implied

lovetox, it’s not about rewriting, it’s about representing in memory or serialised.

And having an easy way to access it.

yeah Link Mauve but thats totally different problem, sorry i cant believe that xml:lang attr suddently is the big challenge of serializing

i see that a server dev, maybe has to change or adapt his serialization process

lovetox, as for “replicating”, in my example the server would only want to copy the body, not the rest of the message. No matter the way it decides to represent it, it must introduce a xml:lang="fr" on the body once copied alone in another stanza.

and in that way, my pervious statement "just add the attr" is wrong

but i dont think this is a problem hard to solve

Actually you have to change jabber:client to jabber:server when sending it over s2s, so that's already messy

also this is all beside the point, users can already add xml:lang to messages, and we expect it to be routed to the destination

are you telling me this does not work currently?

lovetox, it does, but I’m talking about internal server processing (or any entity really).

but im not proposing to change anything here

If the entity wants to extract one element out of a stanza, the parent’s (parent’s (parent’s …)) @xml:lang should apply.

MUC subjects maybe?

what im asking is, that the server sets the attr, not the client, so this change can not lead to problems

Zash, yes, that’s a good example.

Tho Prosody only saves the text there

It should also support setting multiple subjects, each in a different language. :)

My head hurts

In xmpp-parsers I made each text element a hash map of language to value.

Is xml:lang preserved by MAM?

Should be.

Link Mauve: and then you implemented a getOptimalLanguageValue() based on a doubly weighted list from the Accept-Language header?

Ge0rG, only based on a single vector, from preferred to least preferred, defaulting to no lang, or if there is none to the first one.

Without any normalisation (so for instance it won’t pick fr-FR if the user prefers fr).