XSF Discussion - 2020-01-20

dwd: maybe an official blog post? My boss is actually rather reasonable, and I don't think there will be a convincing angle besides of that project I've told you about

Ge0rG: don't know, but maybe this can help: https://wiki.xmpp.org/web/XMPP_summits_for_dummies

Probably the protocol standardization angle is even better. We are part of some such orgs, like IEC 62443

Ge0rG didn't we have a blogpost for your PHB last year? 🙂

Guus: it doesn't explain anything about Summit, merely announces it

one for the iTeam? https://twitter.com/BenBE1987/status/1218983250833608704

nyco they're aware. Someone already brought it up during the weekend (technically, it's not iteam, but some of the same people are involved)

oh sorry

no worries, easy mistake

I'd like to know the reasons why it's not iteam (or in other words, why jabber.org is not run by the XSF)

I bet those reasons are hysterical

IIRC that came up before and there also was something regarding separation of concerns

Historically jabber.org and the JSF/XSF were very intertwined

So it was an active move to separate them, not historical reasons

I'm not sure how much sense it makes for a standards foundation to run a public IM service

something something matrix.org

Does it matter either way?

Yes, if the past few weeks of the XSF have shown anything :)

😲

It's currently running a closed-source XMPP server

noooooo you didn't go there didn't you? 😃

oh dear

XSF = evil confirmed

I meant to demonstrate that it does matter, I don't believe XSF manpower should be distracted from standards work by discussing some random IM service

but jabber.org isn’t the XSF!

MattJ, I’m not sure that one of the largest servers which also has the name matching exactly the trademark the XSF has some power over should be called "random" ;)

but yeah

it needs lots of work to be good

Obviously jabber.org has some historical significance, but it's conceptually nothing to do with standards development (more than any other public server is)

i tried to address some of the LC feedback for 363. https://github.com/xsf/xeps/pull/880 cc Kev, Guus

seen it. I think its an improvement.

wasn't to worried about the previous state, but this makes it more concise.

yes. you were right that it was a weird requirement (someone else pointed that out as well; but i forgot who otherwise i would have cced them as well)

Daniel, The requirement of no access control? I pointed that out. I note you didn't include that as a security consideration; it's implied, and not a hill for me to die on, but it seems worth considering if you're building a system using this spec.

Daniel, More explicitly: I don't think the lack of access control is a bad thing; I do think it's the kind of thing people should be aware of, and perhaps having it in the requirements is enough.

One can still run access control on the network. https://upload.some.lan/foo.png might not be accessible to people outside the network

update https://gultsch.de/files/xep-0363.html

(in section 8.3)

Daniel, Looks good - that's not in the current PR unless I'm being more stupid than usual?

no; i wanted to run that by you before i force push

Daniel, Ah, I see. It looks fine, push, merge, etc.

You can combine this, e.g., with pre-signed URLs, like Amazon S3 has.

At VEON we did something like this: client requests a slot, server returns pre-signed URLs for PUT and GET. Even though there's strictly no access control, as per section 2, it is a *lot* harder to know the URLs.

Another thing you may want to consider is having the links expire after a while. Very short for PUT, e.g. a week for GET.

I actually have a deployment too where the get urls expire. I find that very annoying because I have to go through the service to get them renewed

I can't just put a url in the avatar meta data node for example

Ah, indeed, for avatars, we had something special.

Expiry is quite useless there.

I suppose it would be nice if you could specify desired expiry.

(when requesting a slot)

Which is also useless for avatars

Because you don't know when you'll create the next one. And eternal storage is a nice DoS vector

still, end-users generally seem to expect non-expiry.

I have a simple server-sided implementation. The number one complaint was that stuff eventually disappeared (which was by design).

expiry in the order of months is probably ok, plus a quota

jonas’ I wonder if it is.

if expiry takes that long, people might have grown to expect data to persist.

Guus, or have forgotten about the links ;)

the amount of people that complain will be less, but those that do have an issue, will have a bigger axe to grind.

one can of course also do something smarter than this, but I’d also like to note that in general, clients present HTTP upload as a method to transfer files, not as a method to store things

I don't think so.

then that’s a client issue IMO

you cannot reasonably expect an XMPP service to host files forever

people are used to be able to scroll back in history endlessly, and view media that was exchanged in a chat from other devices, pretty much unlimited.

jonas’ I think end-users do, to be honest.

they don't care about the poor system admin having to pay for storage 🙂

Guus, I disagree ;)

but eh

that’s a pointless discussion then and we’ve had enough of *that*

the bikeshed shall be purple!

let’s boil it down to this: expiry is going to be a real world thing which happens

be it based on time, or based on quota and fifo

what we could do is make it more explicit.

managing expectations goes a long way.

indeed, which is what I meant by saying "that’s a client issue IMO"

again, I disagree 🙂

you want your media to show up consistently on all resources.

I don't think clients can / should manage that.

by client issue I mean: clients should be clearly communicating that HTTP upload URLs are ephemeral

ah right.

which is I think what you were saying

yeah - I was thinking more of clients explicitly negotiating this with the server (which the xep already at least partially allows for, iirc)

We did some calculations on what it would cost to store uploaded pictures and videos. At any significant scale, this is prohibitive.

I.e. if you'd allow indefinite storage.

And sure, I dislike stuff going away quite a bit, but there is so much more media exchange in typical chat exchanges than compared to, say, e-mail, that I think it is very reasonable for server admins to decide on relatively low expiry, in the order of a few weeks.

For avatars, of course, you should be able to select "indefinite"

What's the difference?

Avatars are small? and you only have one of them?

i'll have to limit en cleanup old files on my xmpp server as well

MattJ: if avatars would expire, that be not be great

ralphm, I agree

But we're against indefinite hosting of stuff, and now we're making exceptions - I could upload a 6GB movie and claim it's my avatar

Just trying to figure out what the rules should be

MattJ: what a reasonable size is, is debatable, but it doesn't compare to typical media sharing behavior I witnessed at VEON, or in, say, WhatsApp

MattJ: I could imagine servers offering different "buckets" for different use cases. One for avatars (you can only have one, max this size, no expiry), v.s. media sharing (many, max total size, max expiry)

And that clients have to specify which bucket to use.

_also_, if you are sharing stuff from an existing storage service, like a picture that is also stored in Google Photos, maybe it would be much better to share by reference.

Then, the recipient's ability to go back and still retrieve the image is dependent on that service, instead of your server.

Lastly, if you are concerned about your local archive, maybe servers should provide a way to copy and store media shared with you, and reference that from your MAM archive.

ralphm, re sharing by reference, that’s already "supported" by some clients (e.g. in poezio, you can /embed any URL instead of /upload local file)

ex.:

https://1.bp.blogspot.com/-9OhTPiCkYUo/U5GdYy8atqI/AAAAAAAAWDc/Fj6bsVE2K18/s1600/Well+thats+just+like+your+opinion+man.jpg

jonas’, yeah, so Gajim just shows a link, but Conversations shows the picture.

I installed some Gajim plugins now. That helps.

:)

(and JabberCat will simply OOB-tag any URL-only message)

I know that there's a group that resists any suggestion the XSF should make UI/UX mandates, but I'd really like to suggest renderings for some messages so we have an agreed way to send pictures etc.

What if we'd call them Usability Considerations? ✏

Uability is faaaar out of the scope of the XSF

Also I like the idea of mirroring shared files on the user's server, but the only problem it solves is when the remote file hoster has a shorter expiration than your MAM

Ge0rG, sure. But I control which server I use, and by extension what that expiration is. In my case: self-hosted ⇒ infinite.

ralphm: yes, I understand that. But it doesn't scale to mere mortals

Ge0rG, as for usability being out of scope, I don't see any problem with people writing XEPs discussing usability aspects. Most people writing XEPs are not security experts, but still write Security Considerations. Anyone implementing a XEP still needs to make their own determination on security.

ralphm: that message contained a sarcasm element, but I forgot to add a legacy body. Sorry.

E.g. when specifying Reactions, you can suggest how this can be / is typically rendered, in different types of clients (GUI, text-only, text-to-speech). Or point to a document like this: https://indieweb.org/reacji

Ge0rG: I just responded as if it wasn't, as not every casual reader would be able to detect it.

Ge0rG: Like in Matrix, with some special URI scheme with some touple of server name and file id, retrieved from your own server always?

Zash: something like that. would also close the ip leak to adversarial servers

Server-hosted very strict proxy? Chained Proxy65? Magic?

apt install squid

But with a stable cache.

That persists after the original is gone.

This is a can of worms.

Sure.

E.g. when signing individual messages

But somebody wanted a quick&dirty solution to file sharing.

Is there an existing implementation of MAM that allows for keyword search? If so, what's the form field definition that's used for it? XEP-0313 doesn't specify that functionality (but does list an example 'urn:example:xmpp:free-text-search')

I'm adding something for Openfire. Might as well re-use something existing.

cc MattJ

MAM that ships with Prosody doesn't have full text search

ejabberd might have?

Guus, ejabberd has something, and I have a draft spec in my head

just need the field form definition for now 🙂

The main issue is that you have to do dumb search or specify what operators are supported

Dumb search is fine for some users, but advanced search in e.g. Slack is quite helpful

But they can be separate specs obviously

I've been contemplating learning to program a Lua module to write a MAM module for Prosody.

That uses Elasticsearch as backend.

I'm faking stuff by feeding it in a lucene query

simple keywords will work, but more elaborate lucene queries too (although you'd need to know the index fields)

ralphm, you'd want to implement a storage module, we have an internal API which should map fairly nicely to ElasticSearch

Guus, that's the kind of thing I wanted to avoid

MattJ, are you aware of anyone doing this before?

ralphm, I don't think so... it was somewhere on my todo list but everyone I've worked with has been fine with postgres for now so I never had an excuse to implement it

I also need to figure out a way to sync my Gajim history /back/ to MAM.

can't you just expose SQL?

Free-form WHERE clause?

free-form SQL

XQL?

Mmmmm `UPDATE "users" SET 'admin' = 1 WHERE "username" = me`

urn:irf:xmpp:free-text-search it is!

irf?

ignite realtime foundation

Have you registered that?

Guus, have you checked with ejabberd?

4 minutes of googling didn't turn up anything of interest.

give me a second; i know xml

the field is called 'withtext'

Not namespaced?

nope

withtext works for me.

thanks Daniel

Meh, very non-standard

Form fields that aren't standard are meant to use Clark notation

Yeah, don't invent your own URN namespace.

withtext is not specified in any XEP or registry ✏

MattJ, hm, does that example use clark notation? It probably should

Maybe, but it's an example of what another XEP might define

I'm either going with compatibility with ejabberd, unless someone suggests a semi-futureproof alternative in the next few minutes.

If it's just one implementation going rogue, fine, but if everyone follows it's kinda annoying

Especially since your search syntax *isn't* plain text, but lucene

so '{xmpp:openfire.org}withtext' or whatever you want

MattJ: what does plain text mean, exactly? Every implementation needs to interpret the query string somehow. A few examples: a) does the input get tokenized? b) do the tokens get stemmed? In which language? c) does a space mean AND or OR d) any other processing like case folding

Fields can have a description. Just paste the entire docs for your fts engine in there :)

technically it probably doesn’t matter much; you just take direct user input; and then depending on server users might see different results; but than can be ok

i mean search doesn’t need to interop between multiple servers

it's kinda nice that you get somewhat similar results though

ralphm, What Daniel says. If you're asking for a plaintext search, then the interop point is that you can ask, not what results you get back (which is a QoI case).

Daniel: Sure, but if you want to offer client-side assistance, like how Slack suggests search modifiers, then maybe it would be helpful to figure out how to relay that information.

Yeah, but it's annoying if you're in three different MUCs, and the search on your client yields different results based on the same syntax (eg: spaces result in an AND or OR query)

well you could probably specificy low hanging fruit like "it means AND"; but i don’t think stemming needs to be the same across servers

Guus, How would you tell?

Guus, I mean, unless they were interlinked MUCs via FMUC or something equally esoteric.

dwd: say I'd remember having a conversation, but forgot in what room. I'd like to use the same query on all rooms.

dwd: you can't, so it's even more annoying

Local MUC history?

MIX!

MUIX

Surely any server with search that didn't find stuff in any obvious way would just get fixed?

surely. I'm currently fixing the fixes applied by the previous fixer.

aaanyways

does this make anyone extremely unhappy?

form.addField("xmpp:irf:mam:free-text-search", "Free text search", FormField.Type.text_single);

(other than that it's Java)

Guus, that makes me unhappy

Oh, wait, not urn:

does this make anyone but Zash extremely unhappy?

Guus, I’d prefer {urn:dns:openfire.org}free-text-search or something similar

or `{xmpp:openfire.org/stuff-maybe}free-text-search` like MattJ said earlier

is urn:dns:mydomain a thing?

it is

is it?

til

I don't see it in https://www.iana.org/assignments/urn-namespaces/urn-namespaces.xhtml

ah, dns:mydomain

not urn:dns:

https://tools.ietf.org/html/rfc4501

or `urn:uuid:$(generate-uuid)` :)

or that, but that’s quite opaque

Nice for experiments tho

true

I had a thing that used a hash based UUID over its own source code.

Great for interop

Yeah, either you have the same version or it doesn't work.

urn:dns is silly for this, I think

urn:dns isn’t even a thihng

Just use https URLs

No, use gopher: URLs!

This is *definitely* the protocol developers' equivalent to bikeshedding

Yeah, I'm back at freetext.

Hold on let me find where ASN.1 namespaces are mounted into the urn: tree

Yes, indeed, but using a non-registered URN namespace is just *wrong*.

or whittext or whatever it was that ejabberd did

urn:oid:1.stuff?

Guus, you've had a number of acceptable suggestions, choose one or state why they aren't good enough

xmpp:irf isn't acceptable because 'irf' isn't the address of an XMPP service

Unless you register the .irf TLD for 200k :)

And then still, single-label DNS names are, well, terrible.

Yeah, I'm going for the same as ejabberd until there's consensus.

> 15:36:49 Zash> or `{xmpp:openfire.org/stuff-maybe}free-text-search` like MattJ said earlier

Guus: what does ejabberd do?

There won't be consensus because almost all the suggestions were valid options

All more valid than 'withtext'

any client that currently supports the feature will probably use 'withtext'

ugly as hell, but at least that adds some functional value.

And thus we have another de-facto standard.

With a different syntax

Sigh

Thanks

I suggest the eu.siacs.conversations.withtext namespace.

:sob:

Guus, to be clear, it is XEP-0068 you are breaking by using 'withtext': https://xmpp.org/extensions/xep-0068.html#approach-fieldnames

(and so is ejabberd, but it's not too late to save you)

I'm happy to change it whenever there's consensus on something - a protoxep, or whatever. I've now had various suggestions that were all more or less desirable - I simply don't have the time to evaluate all options today.

Guus: Indeed, please just use clark notation with an https URI.

stuff hasn't been released yet anyway

Guus, the problem with protocols is once you do that, you're stuck with them... need I mention vcard-temp?

Guus: it is more important to not do the things people said are definitely wrong. Including 'withtext' (unqualified), unregistered URN prefixes, stuff starting with 'urn:xmpp:' when the doc is not in our inbox.

MattJ: or even jabber:

> need I mention vcard-temp? WHY WON'T IT DIE? I've been trying to push for vcard4 but everyone just uses more vcard-temp, like adding it to MUC :(

inertia

But then XEP-0084 is sorta supported by everyone and Pidgin