XSF Discussion - 2018-03-06

pep.: there is an alternative development in https://github.com/ge0rg/easy-xmpp-invitation/tree/bootstrapless

pep.: unfortunately, Martin and me did some work in parallel, because I forgot to push 🤦

What hip new framework is that? Is it anything like vanillajs?

Ge0rG, did you mean to ping marc?

jonasw: right, I must've been scrolling too fast.

Sorry pep., I meant to respond to marc indeed.

Redshift?

Ge0rG: thanks for the info ;)

jonasw: is there a tool to maintain xeps/xep.ent? I'm missing &xep0401; for an update to PARS.

Ge0rG, not yet

I can either update it right now or you can do it yourself

please pick one

jonasw: please do so, thanks :)

Ge0rG, I guess I don’t need to botehr with issueing a deferral for XEP-0379 (PARS) then? ;-)

Ge0rG, xep.ent updates pushed

jonasw: thanks very much. I have had some pending minor changes for 0379 that I wanted to push before mid-Feb.

Wanted and failed to.

lets keep the spam low and defer the deferral for now then

Thanks

tbh it’s more laziness than anything else ;-)

that and trying to prevent my domain from getting a bad spam score for all those mails ;-)

Would it make sense to have the same xmlns for 0379 and 0401?

0401: <preauth xmlns='urn:xmpp:invite:1'/> 0379: <preauth xmlns='urn:xmpp:pars:0' token='1tMFqYDdKhfe2pwp' />

only if you want to tie into namespace bumps of either

otherwise I’d suggest not to

but I lack context

Right. That might be unfortunate.

Though I'd like to burn the notion of namespace bumps with fire anyway.

what’s your suggestion what we should be doing instead?

jonasw: feature-flags

and the difference is?

hm, those help in some cases, but they don’t always make sense

Zash: the difference is that it's a better protocol design and that it won't break people's expectations when they upgrade conversations without also upgrading ejabberd.

Ge0rG, how?

the http upload update is a particularly bad example. it changed the whole syntax of the thing

even *if* that was represented by a feature, you’d still most likely implement only one variant.

Hm.

Maybe I haven't thought through this.

It's less terrible once the protocol becomes more stable

it helps with small things; if there had not been a syntax change but merely the addition of support for http headers, sure, that would work fine as a feature. however, the server would also need to detect that the client supports that. requires either caps, an additional roundtrip or a payload in the request.

More options doesn't equal better

jonasw: https://github.com/xsf/xeps/pull/599

I don’t consider the URI change editorial fWIW

but that doesn’t matter for you

I don't even like the URI change.

(it *does* matter in the sense that this would otherwise have been the trigger for a discussion on whether purely editorial changes should delay deferral)

I could be asking for a Last Call instead, but somehow nobody implemented PARS in the last year.

I’m sorry

(I actually am)

I am as well, because I considered this a very effective way to make Jabber user onboarding easier.

Unfortunately, some other widely-used implementation's author objected on formal grounds.

C?

Oh, in unrelated news: it seems like yaxim has over 100k total downloads on Google Play. I'm impressed

lz

(that’s like gz, but different)

So different I'm not getting it. Sorry.

zstd?

Ge0rG, man lzop

jonasw: so while I'm impressed, you are compressed?

Ge0rG, oh dear. so, gz is a short form of a leety "congratz" from back in the olden days when that was cool. or people thought it was cool.

does that help?

jonasw: ah, now I see the arc you are bending. Thanks.

docker hub has the worst web UI ever.

i have no idea why I need to "cancel" a build like three times before it takes effect. the first time the UI just won’t respond at all, the second it’ll change the button to "cancelling" but nothing further happens (after a reload it’s back to normal), the third one is finally effective

not to mention the "X minutes ago" timestamps which only refresh on areload

but which somehow suggest that the view was live

</rant>

Pray to the Browser overlords for magic no-js fuzzy styling of <time> tags

Kev, wouldn’t a nicer way to do that "MAM <before/>+<after/>" be a form field in the query?

it not only nicer, anything else would be in every way wrong

But paging is defined by RSM.

Zash, that’s exactly why I’d prefer to have this in the form. alternatively, define the direction in that case via the form.

and update RSM accordingly

lovetox, regarding "start to sync again from that ID", that might be arbitrarily inefficient.

we already concluded that mam is sometimes inefficient

no need to make it worse :)

RSM has nothing to do with what you want to do

lovetox, huh;

you want to query archive-id 😶 to archive-id = Y

there’s an emoji in your message

that should be defined in the query to MAM, like startime and endtime

that sounds reasonable, but everyone is just talking about using <before/> and <after/> for that

RSM is only for browsing the result

that makes no sense

FWIW, in Prosody, direction is set to backwards if <before/> is present

What happens after that depends on the lower storage layer

Zash, is that even standard?

<before/> gives you the last page of a result

that is standard

MattJ: What, exactly, are you asking?

Zash, afair in XEP-0313 results are always transmitted chronologically

(scnr and just kidding)

MattJ: That's taken care of. This is different.

So you should get a page where the last item preceeds the one identified by <before>

If you set <after> ... then either the results include that item or they don't

There's no voting

Aw

Zash, tell that to council

Zash: FWIW, in M-Link direction is set to backwards if <before/> is present, as well.

And specifying after never includes that item in the result set, regardless of whether you used before or not.

I think that's how most implementations would do anyways? Is there any "black sheep" not doing that 🤔?

(about <before /> going backward from x)

MattJ: the way I read RSM, that's how it should behave. before/ is quite explicit about directionality :)

I'm not promising that all Prosodys storage backends deal with the value of <after> if <before> is also present

> jonasw > Kev, wouldn’t a nicer way to do that "MAM <before/>+<after/>" be a form field in the query? Why? RSM's already defined.

Does RSM define what happens if you include both?

Zash: it does not. But it similarly doesn't tell you you can't.

Kev, I checked the other day and didn’t find a definition of what happens both <before/> and <after/> are given.

Zash: No, but if we're just careful in 313 that we explain it, we're golden.

I'm not suggesting that no text tweak to 313 is needed. Just that we already have the protocol and don't need to invent more.

🤔

Kev, I’d argue anything which specifies RSM behaviour should not be in 313, but in the RSM XEP.

jonasw: I think that, in this case, I'd disagree.

But equally I wouldn't really mind it being in 59 either.

Default file based storage doesn't support both, so results would include the item in <after>.

<before /><after>uuid</after> glitches a bit in my mind <before>uuid</before><after>uuid</after> sounds like in between

And glitches too

Maranda: your first example asks for the last items according to your requested page size, or some server default (might be inifity), but nothing that's uuid or older.

Note that servers are very likely not to allow infinity-sized pages, as that's quite a burden on resources ;)

Is the RSM discussion led in the context of "Dusty Drafts"? :D

intosi: I had to learn the hard way, that even with a limited page size a DB might still lock up the whole server for multiple minutes merely to count the total number of results.

O(n) all the things

Mar 06 14:28:30 mod_log_slow_events warn Slow event 'iq-set/self/urn:xmpp:mam:2:query' took 12.01s: ...

YEs. Counting items is a terrible idea.

intosi: except it's required by RSM

Just as expensive as generating the entire set, usually.

Ge0rG: read closely.

> <jonasw> Kev, I'd argue anything which specifies RSM behaviour should not be in 313, but in the RSM XEP. +1. I don't want an RSM implementation and a separate MAM-RSM variant.

Obtaining the item count is, but including the number of items in 313 is weasleworded.

Obtaining the item count with RSM techniques is

intosi: hm. "The responding entity then returns the item count, which MAY be approximate rather than precise if determining the exact number of items would be resource-intensive"

count=rand()

Just always return 42.

No, return requested+1 if you know there is more.

;)

intosi: Isn't that implied by complete=false

No.

In SQLite, adding a `LIMIT` clause will change the order, unless you have an explicit `ORDER BY` in which case it's still going through the whole list.

That indicates that between the items specified, you may not have returned all items you think there could be in the set.

Ge0rG: depends on which data set you order-by.

Query optimisation is an art more than a science.

intosi: Unforuntately it is. Especially if you have abstracted away DB access through some generic library.

Simple. Never do such a silly thing, unless you own the abstraction as well ;)

intosi: tell that Zash :P

Zash: what I said^

intosi: Tell Ge0rG that SQL things are terrible and why we can't have nice things

intosi: and don't tell me not to use software where I don't control the whole stack, down to CPU µcode.

;)

Because I'd rather spend my time on Easy XMPP

I mean Jabber.

well counting isn't an issue here (as I never operate on the database directly), yet since at the time reading RSM spec I thought a before+after combo wasn't a possibility I don't handle those cases.

I mean Zimpies.

Metronome will handle before but not after if both are present.

Maranda, at least return an error when you can’t handle a case and know it :<

Ge0rG: How much do you care about the total anyways?

jonasw, I'll add something to handle those now.

Zash: I care about my server not lagging to death.

Zash: no idea how much MAM-enabled clients rely on the `count` value

Zash, just exploit 313 and return number-of-items+1

Maranda: Yes, I think it's quite obvious that the idea behind 0059 is to specify *either* <before/> *or* <after/>, and this is just not made explicit because the author didn't even consider the possibility to specify both.

If you disable archive pruning, it stops counting.

So I think we can't just cheat and allow both in 0313.

(Though I clearly see the use-case of course.)

lovetox suggested to use form fields instead of <before/>+<after/>, which I find sane

Holger: I don't think it's cheating, and we clearly can, because we have :)

jonasw: Yes, me too.

You break a MUST in 59 if you include both and the range is larger than one page

> first item [...] MUST be the item that immediately follows [...] in the <after/>

Kev: Well yes we can do ugly things, and I agree we have.

Zash: well spotted.

Zash, only if you iterate backwards, or?

> last item in the page [...] MUST be the item that immediately preceeds [...]

Zash: we can override a MUST with follow-up spec

excellent

But doing before/after in 313 is possible. It's just going to get ugly specifying the same things twice.

Ge0rG, but then clients need an implementation of RSM and one of MAM-RSM

"just"

jonasw: They do anyway, in practical terms.

jonasw: hm.

Kev, do they?

jonasw: And servers.

RSM is very protocol specific anyway.

Kev: Why?

Sorry, not clients, servers. Clients don't.

Huh.

I have an interface in mind which is completely agnostic to protocol which uses RSM by passing a function around.

jonasw: next?

kinda

where RSM would just be a building block to obtain an asynchronous iterator

jonasw: If I were building something today, I'd make it more like a file handle

Requiring server-side state?

and I’d like to keep it that way.

server side state sounds awful

It is.

It would be, yes

(encode the state in an opaque base64 thing!)

MattJ: I have had the thought not limit ranges, and stash the iterator along with the current id, so one can resume from there :)

and intosi, tbh why not just doing <before>uuid-that-was-after-then</before>...?

Cursors are not something you'd want to keep around for every query sent to your xmpp server :)

intosi, that's what glitches

LRU cache or whatever

On the assumption that the client will likely ask for the next page right after the current, it might be worth it

Maranda: I don't understand your question, my head's only partially working.

mine as well tbh lol, but <before/><after>uuid</after> sounds exactly like <before>uuid</before> in my head.

nope

Wat

<before/> is very specific: it's before EOF, which is something that doesn't have an UUID.

You can't specify before last uuid you know, because it will never return the last uuid you know, which may not even be the last uuid in the dataset in the first place.

Hm ok.

you try bend rsm to do what a simple query filter does a lot better

if i can query MAM with end and start archive ids, and my result only contains whats in between

i can use RSM to page forward backwards wahtever

That's exactly what RSM does.

no

Yes

no :)

well, kinda

RSM pages inside a Result

yeah

<{dataform}x><field name=sql><value>SELECT * FROM archive WHERE whatever<///>

it does not filter a result

Zash, good idea

Zash: expecting your protoxep before CoB :)

the Result of the server is the book, and with RSM you switch to a page

what you try to do is to rip site 1-3 out of the result with rsm

and create a new book

No, it's really not.

RSM is specific for a protocol, and we're talking 313, which clearly defines the uses of RSM.

lovetox: Hmm, epub over xmpp!

The part lovetox explained is not specific.

It's exactly the generic part specified in 0059.

If it's all specific then ditch 0059.

...

well, until I understand clearly what to do with both and my brain doesn't kernel panic trying to I just think I'll throw an error.

Throw it all out, define the archive as a giant XML blob and do binary serialized xml delta transmission and reinvent wave?

We should've gone with IMAP from the start.

I really dislike how we keep bending/breaking our specs in other specs.

Holger: Myeah. Consistency is nice.

RSM does a simple thing, and it does it nice

if you want to have more elaborate querys for mam, just define fields for it

like start-id, end-id

lovetox: For queries, yes.

But for getting pages from results, RSM is good at that.

It's just a shame that dealing with before/after isn't well specified.

maybe i dont get it, i request a query from MAM with start-id, end-id

Is there anything preventing us from clarifying that?

what exactly would i need before and after in the RSM page query afterwards?

Given multiple implementations doing something, sometimes, with it

Zash: clarifying sounds like a good idea.

Kev: It's not well-specified because it goes against the 0059 model. You can't take your result book and page forward from page 15 *and* backward from page 30 at the same time.

jonasw, *ducks* actually if before and after are currently present I *already* throw an error.

Kev: Right now you specify a page and a direction. With your change, before/after are no longer directions but boundaries.

huhu

jonasw, both of 'em that is.

Presence or lack of <before/> sets the direction atm

exactly, and boundaries you specify in the MAM query, and not in the paging mechanism afterwards

So I think you can read it as <before>+<after> = backwards but stop at <after>

Kev: And you sound as if it was obvious that specifying both boundaries should set direction=before. This sounds is totally arbitrary to me.

Zash: Ugh.

+1

Or does it explicitly say 'forward' if <after>?

Only before tells you anything about direction, IIRC.

We could ditch RSM from 313 completely and just go for something with the same semantics but different syntax, it just seems wrong to me.

Oo

Oh please no

Isn't MAM done yet?

Are we about to go back to archiving..? 😋😘 *troll intended*

https://www.youtube.com/watch?v=Ysf5o5xOGYE

jonasw: thanks once again for your editorial work today

yw