XSF Discussion - 2020-01-22

Is there a way to do with pubsub (or else?) many publishers many subscribers, but only subscribers see everything. publishers see their own items

pep., Defining fulltext search fully would mean servers would have to implement a full-text search engine entirely - it wouldn't handle, for example, stemming in a homogeneous manner, so we'd presumably have to ban that, which feels undesirable. AIUI, MattJ's suggestion is a strict substring field as well as a "magic" field. I think the threat of beer-buying is sufficient to prevent outright silliness (it also prevents anyone being silly and still claiming full conformance, BTW).

dwd, I don't think we have to rule out stemming

nor mandate it

(for the "plain" search)

But most FTS engines provide an advanced query language, and that's mainly what I want to avoid exposing

But e.g. it sounded like that's what Guus was doing, it's usually the (slightly) easier option

Right, indeed. Your suggestion is a dumb substring search, plus magic. I'd aim for magic first, going that any query language is close to nothing. I'm thinking in terms of tsvector in pgsql, for example.

Unless I misunderstand your suggestion here.

I'm just saying there should be two fields, plain and implementation-specific

running with the postgres example, the plain one would use plainto_tsquery() for example

But then your plain one would do stemming for example. Surely?

Yes

I don't see that as a problem

It defines the semantics of the user input, not what the implementation does with that info

"This is a query from the user with no special operators or syntax"

Now find some messages

Which is different to: > <Guus>  simple keywords will work, but more elaborate lucene queries too (although you'd need to know the index fields)

I'm saying we should have a way to expose the elaborate queries (if that's what deployments/implementations really want), but we should also have a safe option

Safe in the sense that you can just throw some text in there and have a reasonable expectation it will return something useful

This is from last year, but I just remembered it: https://opensourceconnections.com/blog/2019/05/29/falsehoods-programmers-believe-about-search/

> When you find the boolean operator ‘OR’, you always know it doesn’t mean Oregon

Though I think my favourite from there is: > A customer using the same query twice expects the same results for both searches

Ah, I see.

Yeah, search is not trivial.

So if I understand correctly, MattJ is arguing that the second sentence of 3.2 of my protoxep should be in effect reversed, and servers MUST interpret any words or characters as search terms, and not treat them as directives or operators.

I can certainly go that route.

Depends. You only specified one field, and it depends whether you specified the plain or the non-plain one :)

Guus wants the non-plain one, and this draft was primarily for him at this point, right? :)

We already implement MAM search, FWIW, and have for years.

But I wanted to add in a way for the server to convey some help text for the non-plain one

Also we need to deal with localization in the various parts of this

Yay.

and that's not easy - it's extremely likely that the server is only going to have FTS for a single locale

But multiple is of course possible with the right setup, I just don't see many people crazy enough to dedicate the resources to that

Yeah, fine with extending my extension to introduce extensions. I was mostly in my XEP project for inbox and figured I'd knock it this once so as to have something.

I can contribute the missing parts, I've had this in my head for quite a while, but I'm way too busy this side of FOSDEM

I'm not wed to anything here except the beers.

"OR an orange juice"?

You don't have to drink the beer. They just have to buy it.

this reminds me of how the formulas in Excel/LibreCalc are language-dependent, so if you work with multiple locales, you always get it wrong

It makes it impossible to claim they meet the standard by syntax alone.

Kev what fields do you use, and what functionality is behind it? My primary motivation was to re-use existing field names if possible, to have overlap.

You expect me to remember things? :o

{ "search", sizeof("search")-1, XEP313_FILTER_TYPE_STRING, "If specified, only return messages that contain each of these words, in any order" }, Seems to be what we're using.

I think that's a sensible implementation of the "plain" variant

that's what Conversations' local search does as well; and what I would expect a server side search to do as well

define "word"

if I search for arc, will it return messages which contain "search"?

will it return messages which contain "The word 'arc' may be contained or not"?

</rambling-about-how-fts-is-not-trivial>

jonas’, Does it matter?

probably not

jonas’, I mean, does anyone care about replicability of search results between servers? (or between the client's local archive and the server?)

Although there's an argument that it might be useful to have a MAM switch that emits only the ids and not the entire messages in case you already have the data. But that feels like an optimisation for another day.

If you already have the data, can't you just search there directly?

Zash, look at how slow conversations is since it got a FTS index ;)

Zash, Well, depends on how much of the data you have.

Also there's the thing where, given an id, it's tricky to retrieve that message

Zash, It is?

You can request messages before, or after, but not a specific message by id.

Zash, Oh. Well, that's stupid then.

Zash, So you'd have to ask for one after, then one before the result you get.

Yeah

Yeah, that's daft.

Or one before, then the one after that.

Either may or may not work depending on how many messages you have

Inb4 inventing SQL over XMPP to solve this

Zash, Ask for one before and one after concurrently.

Zash, Then it's only 2RTT to get the message you actually wanted.

Unrelated question: with RSM, the direction in which you page through the resultset doesn't affect what's defined as the 'first' and 'last' element, right?

iow the order of elements on a page does not differ based on the direction that you page through the result set?

Correct

👍 thanks

Which makes it funky to ORDER BY backwards, get the results from SQL backwards, then flip them and send them to the client.

Zash, Guus, I'd love to see this written down in xep313 (if it's not already).

Doesn't it say in RSM?

ahh, if so, then i guess that is fine too

I suppose it doesn't hurt adding some implementation note about it. Feel free to PR 😉

https://twitter.com/wire/status/1219745367475933185

> dwd> You don't have to drink the beer. They just have to buy it. > It makes it impossible to claim they meet the standard by syntax alone. I claim encumbrance. You don't know how easy it is for them to obtain beer :p

I claim encumbrance. I reject supporting the beer production.

dwd: interoperability still mandates common wire format doesn't it

re MLS/wire

pep., Ah, yes. I thought it interesting primarily because Wire were pushing MLS as primary marketing. It's more or less finished, but it's got all the heavyweight cryptanalysis to go - roughly at the same stage where some early experimental deployment of TLSv1.3 was happening while till in Draft, about a year or so befroe the RFC was published.

pep., MLS-interoperability across federated messaging protocols? I'd expect that to require even more than just a common wire format

A common data model at least, so you can map into whatever format

flow, You could do text message bridging, though. Depends what the goals are.

depends on where you draw the line around "wire" in "wire format"

or, what Zash says

I wouldn't be suprised if MLS needs to be tightly-coupled with the underlying groupchat mechanism

flow, Prepare to be surprised, then.

I am prepared, can I be suprised now?

flow, In principle, if two members of the group attempt to commit at once it could get weird, and the DS is supposed to impose a strict ordering, but XMPP does that anyway so I don't think anything special would be needed.

dwd, DS?

flow, Also, "Commit?". Easiest to skim the architecture drafts and get a feel for it.

will do

I can probbaly knock together a lightingish talk at the Summit on MLS if there's interest. Not that I'm any kind of cryptographer of course.

I’d be interested

reminds me to put me on the list of remote attendants

and reminds me to allocate a day off

I think Andrew's right, we should use what's already in the most popular XMPP server (although it's 2014 it was added, not 2016) and use MAM search the way M-Link does :)

Popularity contest? I object!

Kev, No idea what you're on about, Openfire's only just adding the feature.

I was contemplating how to put that to words, dwd .

Excuse me, that's the weirdest spelling of Prosody I've seen yet

https://cerdale.zash.se/upload/dHpA6ZKtKtstwlTJ/bild.png

lol

to be honest, I have no clue how many instances of Openfire are running

We have download stats, and update check stats, which give some indication, but that's about it.

Lots in locked-down enterprise networks connecting to Active Directory, though.

in the federated world, not many, I think

probably true

jonas’, Still plenty there; I think most of those doing update checks are likely to be federated.

at least not many seen by s.j.n

so not many hosting MUC services

Daniel, larma, lovetox, any thoughts on a swap over to finally sending 12-byte IVs ? context: https://github.com/siacs/Conversations/issues/2578

Relevant: https://github.com/siacs/Conversations/commit/e38a9cd729bfa44d06beb44859516a1eebbb3c92

(and https://github.com/siacs/Conversations/commit/9af056bb16d7294e427dce2d92944c4d12bd8d0f )

it will probbaly happen with the next minor release (not bugfix)

Siskin and profanity are 'fixed' in master

and we will wait for them to release

aw awesome, going to go ahead and comment on that issue

BeagleIM as well (same library as Siskin), should be released soon-ish (depends a bit on Apple)

cc @ Syndace

thanks jonas’, was involved in that decision

ok

what's the implementation status of bookmarks 2?

After what's been done in the sprint?

yeah probably not much

the prosody module should be working now

converts between all 3 iirc

Converts from both forms of XEP-0048 to XEP-0402 format, and then lets the old form of XEP-0048 read from the same store.

The PEP form of XEP-0048 is only considered for migration, after which it is left unusable.

This should work fine since clients can’t rely on this PEP form working when XEP-0411 isn’t advertised.

Yes I actually think that's fine

I know I was super eager on having migration between old pep and new pep working as well. But I don't really understand why anymore

It is now working anyway. :)

Migration, not concurrent usage.

Yeah. I meant concurrent usage. But yeah it should be fine.

You can unload the old module and then load the new and everything should be ok

Yes.

The new module will refuse to get loaded if the first one is in the configuration file.

(Or loaded.)

Yeah that's cool. Yeah I would like to see a last call on that. Get some more feedback from a wider community and then deploy it.

So for once we could actually do it properly and have a LC before deployment

What about the extensions proposal from Link Mauve btw? did that progress a bit? Maybe awaiting for a PR?

The what now?

The changes to the xep went through

let me grep in the list

pep., which extensions proposal?

yours, to bookmarks2

For stuff like password etc., or else

Ah, the have clients not throw away extensions?

yeah

dwd said he was going to add that to the spec.

IIRC.

ok

I would actually be cool if we could make Draft before Berlin

+1

Then we can put the final touches on the implementations in Berlin

at the sprint you mean Daniel?

eevvoor: yes

dwd: is Inbox a sophisticated attempt at testing how many levels deep you can nest a <message> without getting your computer taken away? ;)

That's a cruel and accurate suggestion.

Really, it's a matter of trying to reuse the result from MAM such that things like MAMFC plug into it neatly.

But it did feel a bit nesty. Might be a better way of constructing it by injecting an inbox bit inside the result, perhaps.

maybe I'm just fed up with trying to read nested messages from one-liner XML dumps from my client and server logs

dwd: I don't have a good idea ATM

xmllint --format became my friend years ago, and has remained so since.

Because yes, reading one-line XMPP stanzas gets worse the deeper they go.

Kev: I suppose I need to add a key binding for it to my vim

Kev: it's double nasty in clients that just dump the raw stream instead of individual stanzas, so that your grep dumps a screenful of XML and you need to find the beginning and end of things to be able to xmllint

I'm not sure I understand why <entry> contains the latest message

I mean the whole message

pep.: so that you can show the last message in your chat list

Are you not going to do MAM anyway right after?

No reason to.

To get more than 1 message yes

pep.: you could implement a thin client that only MAMs when you open a tab

^

Ge0rG, sure, and then I just need to do MAM when you open the tab

Because I will do MAM

What I'm interested in inbox is really just the list, because then I know what to fetch via MAM

It's fairly common when rendering an inbox (both in chat clients and elsewhere) to want to show a preview of the most recent message, so including the most recent message would achieve that (without doing 100/200/howevermany individual MAM queriest to get the latest message for each inbox entry).

So it seems useful to me.

yeah maybe.. probably something I'll have to ignore then

pep.: yes.

I still think that poezio should be a fat client, though ;)

Ge0rG, in any case that message is useless to me in poezio

I'll do MAM to sync up with the last known id

The whole fat client/thin client thing I think is only going to be 'resolved' by allowing for both.

Kev: I agree ✏

In cases where allowing for both is going to mean lots of data being sent that one or the other doesn't want, potentially shoving a bool on a query to exclude the noise might make sense.

I actually have a use-case for both. I want a "fat" poezio on my colo server, with full local logging, and a "thin" MAM-backed one on my laptop when I'm on the go

https://modules.prosody.im/mod_map.html

I don't know if that would add any value to inbox or not, but it's a possibility in general.

dwd, had you seen ↑ ?

Zash: Is that also similar to the unread stuff in bind2?

Ge0rG, both can use MAM

pep.: sure, but in different ways

Kev, yes, it's inspired by that example in bind2

pep.: I want my fat client to do a full MAM sync on startup, and then no more MAM

Where inbox is also related to the unread stuff in bind2 (but none of them quite the same)

Ge0rG, when joining a new channel

startup = new session

pep.: history fetch is often good enough, but yeah, okay

Zash: I wonder if there's a race there, by not doing it during bind, but it looks useful.

Zash: Submit a protoxep?

I do think that server-side tracking of unread per-contact is practically needed, which that doesn't quite do, so it's not a whole solution, I think, but is moving in that direction.

Kev: It's mostly done like that to allow easy testing since I don't have bind2 yet.

Yeah, that one's a bit of an issue :)

as is IM2?

Everything2

XMPP 2.0

Zash, I had seen it, but then forgotten about it.

pep., And yes, you might not always want the entire message, and instead just know there is one with a particular id. Or you might not need inbox at all if you're going to pull the entire MAM archive across anyway.

pep., But lots of existing clients like to list out the conversations, and show a previewish thing of the last message. It's why, for example, Instagram's direct message inbox works in exactly this way.

pep., We have bigger challenges because we have lots of different styles of client in the XMPP world, and need to cater for them all efficiently without precluding any. I'm not trying to claim this is a finished design suitable for all cases.

but it's a very good start

dwd: I think it's missing a notion of "open conversations", which is a good thing to keep around in just this place

dwd, my goal is not to pull the entire MAM history

At least not at first

My goal for the inbox thing is really just to get a list of JIDs to fetch MAM for. If I don't have that then I have to fetch then entire history to know who talked to me as there might be JIDs I don't know of (not MUCs nor roster)

pep.: how is having the last message in the response harmful to that?

Ok it may seem I'm still ranting about that, I'm not

Timestamp and body of last message per contact gets you most of the data you'd need to show a list of recent conversations and can be done with simple MAM. Read status needs more tracking than what at least Prosody has

Zash, that's the thing, you might not be talking only to contacts

s/contact/"with" in MAM terms/

yeah but you need to know who, which is why I like inbox

That MAP thing did that iirc. Wanna be convinced to convert it into mod_inbox? :)

Do we need some XPath-ish MAM search thing like the other example of extended search forms?

would it be possible to make that message optional maybe?

dwd, ^

Sure.

Zash, XPath-based MAM search? Yuck.

pep.: what's your goal with that?

Ge0rG, why are you fighting it that much? That message is not needed in there all the time :x

pep.: I'm not fighting, I'm curious. Every boolean options doubles the number of states you create and have to debug

we're at the protocol level still, I think we can live with one or two more options. We're not doing client UX

pep.: please tell me why that Carbon message isn't displayed on my desktop client.

(yes, this is a protocol question. More than UX at least)

even if that makes things more complex I'm of the opinion that I should be able to choose. if we do one-size-fits-all nobody is going to be happy, or rather, only the golden use case is going to be happy and that's annoying for everybody else

So you want to make it optional to request or optional to generate?

Because making it optional to generate would be bad

how bad?

I was mostly thinking "I don't need it, the server doesn't need to send it". Whether it generates it or not (or stores it as is) it's not my problem

What about deployments without MAM? (e.g. for privacy or resource constraint reasons)

with offline messages?

In any case if the server doesn't keep messages, then it doesn't make sense indeed to force it to return the last one

My point is mainly that you may want to support inbox on a server that doesn't store messages. It seems to me it would be easier for client devs to deal with no message than no inbox

Er, I think I'd be fine with "if the server/user has a MAM archive enabled, you must do this"

Just not with making a hard dependency from Inbox to MAM for deployments that don't want that

For something that is ultimately a convenience/optimisation feature

So such a server would always return an empty list?

Ge0rG: I think <entry/> would just be empty

MattJ: what JIDs would you list?

(which is already valid per the XEP)

I guess the XEP doesn't really specify what JIDs are in the list

With previous PEP-based proposals that was trivial, and the list is a list of open tabs/chats

yeah I also liked that

Opening/closing a chat cleanly mapped to adding/removing from the list

Now it's a bit ambiguous

May be it can be resolved by adding another flag to the inbox, one that reflects an open chat and is sticky even when there are no unread messages?

MattJ, I'd want a per-client list though :x (or profiles or whatever. I guess per-client is already good)

pep.: if you have it per client, you don't need to sync it to the server

Ge0rG, maybe it's not just a dumb list in PEP that I want. I also do want to know if MAM/offline stuff that somebody that's not in my roster talked to me and that I need to do MAM with it

if I don't have that information right away, I need to fetch the world and I want to avoid that

pep.: do you want that for all remote JIDs or just the ones that your client hasn't heard from or the ones that are new since the last MAM fetch from any of your clients?

I'm trying to determine which sets of information we need for the different use cases and how they overlap

Why not a special PEP node plus an iq that performs a query basically equivalent to what Dave's proposal has

For the set of JIDs currently stored

......plus unreads??

Why not have both in the same IQ?

Both what?

Both the open tabs and the inbox

That's basically what I'm proposing, yes

So we were misunderstanding each other all the time? Because that's what I wanted all along as well

Dave's current proposal appears to me to not define any logic around which JIDs should be included in the result

I'm suggesting we merge the old PEP inbox proposal, and use that as the list of JIDs, plus include any others that have pending unread messages

So you have a single query for all "open" chats and unread messages

I think it's similar to what you/someone suggested earlier about a sticky bit on the JIDs, it's just not clear to me how that would get set, how notifications would get broadcast to other clients on update, etc.

I think PEP is a good mechanism for that part

And that solves my issue too... clients/servers without MAM can still "implement" the open chats part (PEP) without needing to implement the magic query

Ge0rG, I want a list of open tabs (1:1/muc/whatever), but since I'll most likely want a different one per client I can indeed implement it locally, and also I want to know who I have to fetch when I was offline, without having to sync the world ✏

Also.. the last message is probably not useful for some e2ee mechanisms (PFS).

Ah nevermind.

That would be an unread message :-°