XSF Discussion - 2021-01-04

MattJ, thanks for the reminder

https://matrix.org/blog/2021/01/04/taking-fosdem-online-via-matrix

I suppose it's time to start poking the SCAM team about FOSDEM?

Zash: sounds like it's a month late

Probably more interesting to poke scam about the summit

I personally don't care much for fosdem. But some sort of summit would be interesting

👍️

``` compliancer$ ./compliance xep-0443.xml ~/src/dino/dino.doap {"name":"Dino","type":"client","badges":{"core":"advanced"}} ``` Web enough for ya?

https://share.zapb.de/0e13a6d6e89e55a5708adf142c700a6b2c084b73/iXdwnPOhiCvOnS55xjQRAga0qWwAPURI4yRk1izL/client_data_exchange_poc.webm

Client data exchange PoC from Gajim to Dino

Any thoughts on that? Any chance we can agree on some format and deploy such a feature?

What does that mean? What am I looking at?

Oh, the import/export discussed a while back?

probably just get a couple client devs to agree and go for it

Zash, ye

yes

The XML police would like you to either update the PIE XEP or namespace your new stuff ;)

marc, conversations has a backup format https://github.com/iNPUTmice/ceb2txt

moparisthebest, it's actually an exchange format between client which can also be used as backup

+s

probably the same thing I guess

moparisthebest, no, conversations just dumps its sqlite db afaik

you cannot import this data in dino or whatever client

at least not without writing code

moparisthebest, ;)

moparisthebest, you get the idea I guess ;)

Zash, it is namespaced?

even though this is no necessary at the moment ;D

marc, I mean in the pedantic sense that your new stuff isn't defined by the existing XEP

My xmlnsense itches

Zash, ah, that's what you mean :D

marc, and since I'm only complaining about such pedantic things, you should read into it that it seems fine so far ;)

Surveying client devs might be a sensible thing. And post to standards.

It would be nice if you could shove whatever this is into private XML or somewhere and use it to setup clients on a new laptop or wherever as well as export to move between clients.

So you'd download it on first login and now all your settings are migrated.

Then again, maybe most of this is stuff that would already be on the server so it wouldn't matter.

and to be clear I think it's a good idea too, I was just pointing out an existing single-client impl, and that "backup" and "export that I can import into another client" are the same thing :)

SamWhited, the log might contain senstive data

think of E2EE :D

You could leave those bits out of the server version

Relatedly, XEP-0227 ought to be extended with MAM support

Or encrypt the whole thing before saving it

Encrypt the world and save it on the server.

if it's not going to restore e2e messages then mam already has that use-case covered?

just pull a whatsface and upload all the e2e messages in plaintext to apple and google clouds

Yah, I don't fully understand what this is actually migrating so I'll be curious to see the eventual spec

yes, when everything is in mam it hasn't no real advantage

Is that all this is then? History?

SamWhited, history / logs, downloaded data and client settings where applicable

What are logs in this context?

Messages (already decrypted in E2EE context)

Basically everything stored on a device

In the end it should also be possible to migrate the omemo stuff (private keys)

marc, tricky re: omemo though, you can only copy/reuse the old keys if you pinky promise never to use the old client again, how are you going to handle that?

Client settings seems like it would be very client specific and couldn't be defined easily, but either way it seems more useful to upload those to the server and fetch them the first time you log in

this sounds like something equivalent to Matrix's encrypted key backup

E2EE encrypted messages and downloaded data could potentially be nice to transfer directly between clients without redownloading tons of huge images though

which lets you stash versions of the OMEMO ratchet server-side encrypted by a passphrase

There's probably a number of common settings that might make sense if they can be restored into more than the originating clients.

I wouldn't transfer the keys, a new client is a new identity, but just having message history is nice.

maybe the "export" has a "delete my omemo keys from this client" flag

Zash, +1

moparisthebest, the scenario is where you move let's say from conversations to siskin

Because you don't like android anymore

In this case you won't never use the old device

right, in that situation you *can* move and keep your old keys, but if you get a new laptop and intend to use gajim on it and your old one...

sounds tricky UI-wise for users though

moparisthebest, remove the account from the device after "export with omemo"

I feel like people are going to click taht button, say "wait, it says my keys have changed!" and then re-import the backup on both devices and still have screwed up encryption

or, trying dino side-by-side gajim on the same machine

well, you don't have to use / implement this feature

but it makes sense in the backup case

and device migration

which is quite common for mobile devices

I like it, I'd use it, I'm just pointing out hard problems that need solved :/

sure, you have to warn the user or make it an export option or don't implement it at all

But it could be specified IMO

you probably need to warn them on import (too?)

As optional or whatever

as SamWhited said, they could export once, and import the same backup on 2 different devices

Just because something can be specified doesn't make it a good idea :) I'm not sure if this would be or not

SamWhited, well, for backups that's essential

Today users make backups my copying the .local directory which can be used multiple times as well, so what?

I'm not sure that it is; new devices are a new identity even if you backup everything else

Or no backups and their keys change every time ôO

Users *can* do that, again, I'm not sure that it's a good idea. I'm not sure that it's not a good idea either, it just needs to be thought very carefully about and not just done "because we can" IMO

Yes, that needs to be discussed

larma, lovetox, Daniel, Ge0rG: any chance we can deploy something like that in all (tm) clients? What are your thoughts?

Having only skimmed the last messages, this all looks like the kind of idea where 90% of the specification would involve Security Considerations.

Exporting and reimporting ratchet state and client init keys (prekeys?) all feels like the kind of thing where "HERE BE DRAGONS" would very much apply.

but if you avoid that and leave the transfer out, it's just a backup format that multiple clients happen to implement

moparisthebest: you still get your E2EE messages in your history, you just send/receive messages as a new identity

moparisthebest, What can you gain from this that you can't gain more safely by having a longer-term, more protected (possibly hardware or offline) identity key which signs the client's identity key?

sorry I mean, I think the key *should* be left out, and that "just a backup format that multiple clients happen to implement" is a good thing

oh, right.

If we want keys to be easier to trust maybe it makes sense to have a way to trust keys on your old client when you add a new one and have that trust conveyed to your contacts.

Presumably that would exist in the OMEMO spec somewhere, although I haven't read any of the more recent versions so maybe such a thing exists

Any of y'all following MLS?

yea I think, not sure if it has a name, some type of "transitive key trust" is probably the best solution and negates the need to ever transfer keys

but that surely is full of all the dragons dwd mentioned and more :/

indeed

If only we had one key per user, not one per ratchet click.

PFS for IM is like attaching a sled to a stinger rocket for some fun

marc: a common backup format would require all clients to use (roughly) the same database schema, which is not really achievable after the fact

You could probably define a minimum viable format for plaintext message logs, but good luck beyond that ✏

Ge0rG, can you give an example were do you see problems?

+h

marc: most clients have a concept of a chat list, where details about a conversation are stored, like last read message etc

Those details won't be portable I suppose.

Ge0rG, yes, and if you would like to send typing indicator etc. why not?

Exporting all the image files is non trivial as well. You'd need some mapping of file URL to Chat message to local files

Ge0rG, I already use fsf for that

the poc I posted above already exports/imports shared images

marc: yaxim will process each incoming message in a way that doesn't allow to recreate the original xml.

Ge0rG, not the original but an xml that might be sufficent?

Maybe yes. As I said, the message history is the easy part.

Except that I don't have three different message ID columns, so deduplication with a MAM archive is going to fail probably

Didn't look at the yaxim db but dino and gajim look quite good

Also xml is a horrible database format

Well, I could probably hijack the message sending and receiving functions to generate and consume xml files.

But what do you do with 0184 receipts?

In yaxim it's just a bit on the message row

Do I need to generate a fake receipt message to store along the original, with sender and receiver JID swapped?

I would say yes atm

LMC already works

LMC will just update the message row, I don't keep full history

Ge0rG, xml because if it's not xml everybody says "i don't like [insert format] because i use [insert other format]"

And you already have the parser for XML

Are there clients who use the subject header of an XMPP message?

eevvoor: you can send type=normal messages that are like emails

marc: how do you store a gigabyte of images in xml?

Ge0rG, it's only xml for the messages etc., of course

xml + raw files and maybe compressed with [insert an archive format here]

Ge0rG, and which cleints do show them differently than a usual chat message?

Do you backup avatars? Disco caches? Pep nodes?

eevvoor: maybe pidgin and Gajim?

Ge0rG, atm it's not about "is every client prepared for all backup feature corner cases" but "are there engouh client (devs) that would integrate it"

thx Ge0rG

marc: what's the document format? One xml document with a <messages> element containing a list of all your conversations as regular xmpp message stanzas? One list per contact?

Ge0rG, atm, it's one big file but that won't work for large data sets. maybe XInclude or multiple files, one per contact or so

xep-0227 uses xinclude, for example

marc: I'm using smack as the xml stream processing middleware. I have no idea how I'd plug a file into it.

I'm sure xinclude is a security nightmare.

Ge0rG, yes, I thought the same :D

I still have nightmares from reading the xmlsig security analysis

Ge0rG, I don't know smack very well, maybe there is a way to feed an external xml stream

Maybe

Also what's fsf?

eevvoor: mcabber displays it if it's set

SamWhited, that is cool. Even a CLI client.And can I also send it with mcabber?

Is there a client table which lists this and other features? ✏

eevvoor: yes, you can set it when sending messages in mcabber using a command

perfect

eevvoor, not for this (as it is part of the RFC), but with the granularity of XEPs you now have this: https://linkmauve.fr/extensions/

Which I’d like to push to get merged in xmpp.org soon. :)

Ah your own table! Congrats and thx for putting it together! Link Mauve ✏

It’s not mine, it’s client devs saying what their client supports in a DOAP file. :)

I see. And where do I find those DOAP files?

They’re linked in the "doap" entry of each client in this file: https://github.com/xsf/xmpp.org/blob/master/data/clients.json

I say client, but there is another JSON file for servers and another for libraries, in the same directory.

And those projects should also provide one!

I always vaguely think I should do that, and then I look at the file and realize I'd have to learn what DOAP is and how RDF works and figure out how to actually render XML properly when half the libraries screw up namespace prefixes in different confusing ways and it's all completely unreadable and then I give up.

I should probably give it another go at some point though.

SamWhited, why render it yourself?

Link Mauve: as opposed to what? Is there some service that will generate this for you if you put in some info?

PulkoMandy wrote a XSLT script to automatically present it in a web browser, and eventually the goal would be for the various websites which want to use information from XMPP projects to fetch it from there.

https://github.com/pulkomandy/xmpp-doap

Sorry, I said "render" I just meant "generate XML that's compliant somehow"

Oh.

https://dev.gajim.org/gajim/gajim/raw/master/data/gajim.doap here is the DOAP for gajim.

Yeah, that’s different indeed. :)

Adding XSLT to the mix does not make me desire to learn all of this impossible to read stuff more :)

I would recommend to start from an existing one, like the one eevvoor just linked, and change it to suit your library.

You can ask me to review it once it’s done.

SamWhited, you could qick and dirty just copy paste and adept gajims DOAP.

ah Link Mauve was faster ;)

Thanks for the offer of review; that's probably what I'd do to start, but if I don't automate generating it I'll never keep it up to date

Do you have like one module per XEP in your library or something like that?

Uggh, all of this just seems unnecessarily complicated for no reason though

I’d welcome any suggestion for making it simpler.

Don't use XML, or tons of namespaces, or RDF files and XSL files and XML files and etc.

Maybe basic XML would be fine.

Basic XML, for this purpose, would be something like AppStream?

I don't know what AppStream is

hehe

https://www.freedesktop.org/software/appstream/docs/chap-Quickstart.html

I considered it, but RDF seemed much more extensible, for our purpose.

Why do we need it to be more extensible?

And also already supported by many other consumer projects, which is a good point I guess.

SamWhited, in our case, I needed to have the information about specification support.

I suppose; it seems like I just have to learn like 3 diffeerent technologies just to make a list of XEPs now.

There is no existing format that I could find which exposes this information in a machine-readable way.

You can also just consider it as normal XML and copy/paste from an existing project. :-°

Like I said, I'll never keep it up to date that way, especially when it's this complicated. Or I'll just screw up all these namespaces a bunch.

Anyways, maybe I'll give it another go at some point, I'm just sick of the XML community trying to make every simple problem solved with 5 layers of different XML tech even when it's not necessary.

Sorry, I'm just ranting at this point though. Done now.

You could also make it use JSON-LD I guess, but that’s not simpler.

It represents exactly the same information.

Using {}: instead of <>=, which is a plus I guess?

I don't know what that is either, but I don't think we need anything beyond simple XML or JSON (probably JSON because this does seem like simple key value pairs which is what JSON is best suited for, but either way).

Or TOML for all I care. It doesn't need anythign complicated.

You can write your DOAP file in JSON-LD, there are a multitude of tools to convert between different RDF formats.

SamWhited: look you have to accept XML into your heart as your lord and savior, the one true data format, would you like a pamphlet?

<project><name>thing</name><xep>1234</xep><release><version>1</version></release></project>

FTFY

SamWhited, that’s almost what DOAP is, tbh.

Except I also have to conform to whatever RDF is, and also figure out what all these namespaces mean, etc. I have no idea how I'd even figure out what most of this stuff is

Although with some other useful information, like if this XEP is supported, is support complete? Which version are you implementing? (super useful for keeping track of changes) Do you have some note on your implementation?

Ge0rG, sfs, xep-0446

Sure, I waas not suggesting those were the only things that needed defining, just that it could be a single document in a single namespace without a ton of extra cruft.

SamWhited, RDF is a way to represent a list of <subject> <verb> <object> triples, which is really useful here. :)

Or, as much as you joke about it being {} instead of <>, I think JSON would be a better fit here because it doesn't have all of XML's extra crap which is great if you're defining a giant chat protocol but not so great for representing a tiny handful of key value pairs.

Why the hell would I have to learn a whole new technology and deal with that extra layer of abstraction just to encode a tiny bit of data?

That's my whole point, this all just seems like overengineering for no reason

As I said, if that’s a plus for you, it’s trivial to convert RDF from its JSON representation to XML, or to Turtle, or to whatever.

Anyways, sorry, didn't mean to rant, guess I'm not trying this again because I"m already mad at it again and I haven't even tried to figure it out

It's not about the format, it's about the fact that it uses RDF at all. If I have to learn a JSON equivalent of a bunch of random crap it's just as bad as doing it in XML

SamWhited, the main reason I didn’t “just” invent my own XML format (I originally started that way btw), was that it has a ton of existing tools around, which I would have to build myself instead.

SamWhited, but you’d have to learn something anyway, if you want it to be compatible with other people.

Why does it need tools? It's a tiny amount of data that doesn't need anything special to represent it

There is no magical format that everyone already knows.

Of course, I never suggested there was

It needs tools because otherwise you have a file.

That's all we need: a file.

Just a file, sitting here in your repo.

Many projects already had that, a file.

I mean, you'd have to parse it, but presumably you have an XML parser already. You don't also need an RDF schema or some nonsaense

If not a RDF schema, maybe a… XML schema? :p

I could write you a XML schema representing such a RDF file, if that’d make you happy.

That way you can validate it and generate a parser for it and whatnot.

That is not at all what I'm suggesting

I don't want to do any of that, I want to copy/pate the gajim one, read it, and know what's going on without spending hours learning new technologies just to write out some simple metadata

Have you tried doing just that?

SamWhited, but that’s doable right now, all the items are quite self-explanatory

Ignore the few xmlns, focus on the element names and their content.

Yes, multiple times, and I've mostly given up trying to figure out what every random thing in this complicated files was or why there are multiple xsl files related to it somehow or how rdf works and why there's an rdf file that's apparently an OWL schema, whatever that is, and etc.

Literally nothing about this is self explanatory.

Why are the XMLNS's there if they can just be ignored?

because they explain to machines what this file is

SamWhited, why are there xmlns in XMPP?

I understand what namespaces are and why they exist in XMPP, I don't understand why they exist in this file that's just encoding a trivial small amount of data.

and the few things I find not self-explanatory without domain knowledge in this file, would be the xmlns and the foaf stuff

A <message/> also encodes a trivial small amount of data, much smaller than such a DOAP file tbh.

Oh gosh, yah, there's also foaf, yet another random external thing to learn

(but even then it is quite understandable)

I mean, foaf:Person, foaf:name, foaf:homepage, that is quite clear, especially with an example file

Gajim’s DOAP doesn’t use foaf, the xmlns could be removed.

I mean, we can argue about whether namespaces are useful in XMPP too if you want, but it's a false equivalency, XMPP is not the same thing as just encoding a list that a website can consume or whatever

What, so foaf (whatever that is) isn't even necessary? Again, how am I supposed to know that just looking at one of these files and copy/pasting? Do I just get lucky?

marc: how is that any good for referencing a local file?

I don't get why XML people think that this is somehow self explanatory

Anyways, I'm all mad at XML now as usual. Going grocery shopping for real this time.

SamWhited, it’s necessary if you want to include author information, Gajim decided not to, so they don’t need it.

You don’t need to include information you don’t want to, you could just have <Project xmlns="http://usefulinc.com/ns/doap#"><name>Gajim</name></Project> and that’d be a valid DOAP file.

Any additional information is optional.

Of course, if you want it to be useful you may have to provide it, otherwise it won’t be queriable.

Link Mauve, though I think SamWhited has a point that we could easily make a doap generator for all the fields for people who don’t like editing text files

I started writing that at some point, then realised the bulk of the job would be in maintainance, not in writing it.

(generator/editor)

And tbh, I’d expect someone who is able to write a XMPP software to be able to copy/paste a XML file and change a few fields.

Link Mauve: heresy!

Link Mauve no one simply write XML by hand ! A true XML library you need