XSF Discussion - 2021-05-04

I'd like to support PubSub nodes with an unlimited number of items. AFAICS there's no `max_items=unlimited` thing. We now have `max_items=max`, but I think the meaning is a little different. (Plus I'm unsure whether there's an agreement on whether `max` is going to stay that way, now?)

And so it begins

max was as many as the deployment will allow, wasn’t it?

So my idea of how to proceed is: I'll submit a PR that adds support for `max_items=unlimited` to 0060, everyone's thrilled of the idea, it's merged this afternoon! That's how things work, right?

Kev, yeah.

Which, if a deployment allows unlimited items, would mean unlimited.

And if the deployment doesn’t allow unlimited would mean as close to unlimited as you can get.

So I guess the _idea_ was something like "client configures max_items=max" and sees the resulting limit when querying the config from the server.

Where "the resulting limit" is an integer.

Not necessarily (IMHO)

Or is it fine for the server to return `max_items=max`?

IIRC yes

Only the server would know the true limit

Might be two questions: (1) is it fine as per the wording, (2) is it fine as per the author's intention.

Given that that value may change over time (based on configuration, implementation default, etc.)

Clients would just see 'max'

Ok. Works for me. Kinda even better than my idea of how to proceed 🙂

So people were fine with the `max` thing, or at least gave up on going `-1` or something instead?

I.e. if I add support, there's a slight chance it's still in line with 0060 next month? 🙂

There's a significant chance

\o/

Where's that PR?

I think we decided it's probably easiest at this point to add a new type for this field

(and similar fields in other XEPs)

The `max` thing is already in 0060.

🙁

Holger, https://github.com/xsf/xeps/pull/988

I just was unsure whether it (a) can be (ab)used for my use case and if so, (b) whether it's there to stay.

Ah.

MattJ, thanks.

"unlimited" will throw a spanner in the works

Heh.

I'll shut up then 🙂

And work with `max`.

I think in my mind "max" was "unlimited" (or as high as is possible)

which is why it doesn't make sense to translate it back to a number

Hrm.

It's unfortunate that the client can't then know whether it's actually unlimited

Strictly speaking I would still think it does make sense.

Right.

It both does and doesn’t.

You're right and wrong

I agree with 50% of that :)

A disco feature for max-means-unlimited might make sense, I guess. Or some way of making the max discoverable (which I don’t think it currently is, but might misremember)

I would still think the proper solution is to have both `unlimited` and `max`. They mean different things. Having max-means-$whatever in disco instead sounds weirdo to me.

Having both seems weird

The two reasons that come to mind for it making sense to convert back to a number for display purposes (ideally while persisting ‘max’ internally) are that ‘max’ will break unsuspecting clients, and that it’s nice to know what the max is. The rason for not converting to an int is that you want the client to know it’s pinned at the max.

MattJ, why?

If unlimited exists, why would anyone ever use max? Unlimited is higher than max, or at least implies there is no maximum

Or, why would anyone ever use unlimited, I guess. Because you probably want either unlimited or as close to it as you can get, which is what max gives you.

There's either a limit, expressed as a non-negative integer, or no limit, expressed as `unlimited`. The client won't know. So we have the special value `max` which means `unlimited` or, if not possible, the highest limit allowed.

Right.

*Unless* you want unlimited and anything less than that you want to error on.

The whole "max" thing was added because clients wanted to be able to shove as much into the node as they could, without picking an arbitrary number ✏

Right.

Which is exactly what you're after with "unlimited"

Yes.

And we couldn't have advertised the server limit because clients want it pinned to the max, even if it changes

In my specific use case that's actually what I'd want.

(Return an error if `unlimited` is not allowed.)

I don't think it makes sense for a server that allows unlimited pubsub to also have an arbitrary "max" value

Holger: But in that case, the absense of a “max means unlimited” feature would work for that, no?

I can live without that error, but it would be the proper solution.

Kev, yes, I'm trying to argue for the proper solution being to have both `max` and `unlimited`. In which case max would of course not mean unlimited.

So we just shove more enum entries in that?

Why not `default` too‽

In my book, yes. enum entries are relatively cheap, no?

Especially when mixed into the same data type as abitrary integers

> I don't think it makes sense for a server that allows unlimited pubsub to also have an arbitrary "max" value I don't quite understand.

> In which case max would of course not mean unlimited. No? I’d have expected when ‘unlimited’ was supported for ‘max’ to mean ‘unlimited’.

Kev, ah. Yes 🙂

Holger, what's not to understand? What would "max" be if the server allows "unlimited"?

Seems I'm having a hard time to express myself.

Client developers that pushed for "max" didn't want it because they wanted some arbitrary limit

Welcome to the Internet :)

If the server allows unlimited and the client requests `max` then the result is unlimited.

From the clients perspective, if they configure with limit=unlimited, then if it's supported, everything is ok, but if it's not, then they get a proper error.

They pushed for it because they wanted no limit, but aren't necessarily going to get that

Max was certainly intended to mean unlimited (from a client's perspective)

^

Eugh

> Max was certainly intended to mean unlimited (from a client's perspective) With the realisation that some arbitrarily high limit might exist, because unlimited storage hasn’t been invented yet, except in /dev/null, right? :)

Daniel, my assumption was, for the case where the server allows 1000, `max` means 1000.

Holger, that's correct. But such a server wouldn't also allow unlimited entries. Would it?

If it did, then "max" == unlimited

What would probably be neatest would be for a server to return an annotation when a client configures the node to ‘max’ saying what the effective value of ‘max’ is, with ‘unlimited’ being a possible value :)

Such that a Holger client can abort because 9,223,372,036,854,775,807 isn’t high enough, but everyone else can not care :)

MattJ, I don't get it 🙂 If I configure my server to only allow 1000 items for a given node, then this server won't at the same time allow an unlimited number of items for that node, no.

But you obviously meant something else 🙂

My wishful thinking was that what ever internal limits the server might have it won't matter in practice because they are hight enough

So max would be a de facto unlimited

Holger, no, that's exactly what I meant

Just pretend `max` means unlimited and carry on :)

So if you guys are telling me to just use a super-large integer in place of `unlimited` because it has the same effect in practice then I at least get the idea. But no I'm not going to implement `if limit < 9,223,372,036,854,775,807 then delete_old_items()` 🙂

MattJ, yes yes, works for me.

Are you talking about the client or server perspective here?

Server.

I'm not sure why the server needs to care. Either you have a limit, or you don't

I'd like to allow clients to configure nodes without limit.

If you don't, then max really does mean unlimited, not some arbitrary value

limit=max → high as it goes plz limit=unlimited → give me unlimited, or give me ERROR

Hm

Wouldn't adding entries to this enum require yetanotherxeprevision?

Yup. I totally get how this solution works in practice. Assuming it's fine to return `max_items=max` when the client asks for the node config.

Which would error on most servers, because unrestricted resource consumption is unrealistic for most deployments :)

Aaaaaaaaah

Holger, it is

I'll insist that strictly speaking, it is hacky and wrong, but it works for me 🙂

It would be wrong to not return "max" in the node config

Whether you have a limit or not

Wait, you're saying I should return `max` even in the case where there _is_ a limit?

Yes, because that's what the configuration is set to, "max"

So for a node with max_items=1000. If the client configured `max_items=1000` I should return `1000`, if the client said `max_items=max` I should return `max`?

Did we bump the xep60 config namespace for this? Because if not, returning max to an unsuspecting client under the previous namespace that only supported ints would be wrong :D

Kev, sure, it broke Prosody assumptions about it being an integer too *shrug* :)

Why do clients get a namespace bump but servers don't?

Becuase the server would reject max as being invalid until it supported max?

Holger, IIRC 'max' should only be send by the configuring entity and the real integer value is returned by the pubsub service

but I could be wrong

(Although I note that there are definitely deployments that are doing checking against schemas (not necessarily XEP schemas), and so any changing of protocol like this without a bump breaks them, and that’s bad)

and I woudldn't be suprised if this is not clearly stated in the XEP

flow, that's what I assumed, and then everyone told me I was wrong 🙂

(Checking about schemas on the way through a routing entity, that is)

Right, it's not clearly stated.

Holger: I think only Matt said that’s wrong :)

And why can't we do nice things like advertise the the types in the config form? ✏

pep tried to get the types in data forms validation, but that processed stalled

There are no schemas, since it's a data form

and it's text-single

Kev, ok, and admittedly he only said half-wrong half-right, but in the end, wrong, or something 🙂

Holger, I am curious: why do you want unlimited? as Kev pointed out, the inifnite tape of a turing machine is a theoretical construct ;)

MattJ, dataforms validation is sorta like schema tho?

Zash, yes, but not used by XEP-0060

Nice things, can not have.

and as server dev, I'd expect you to be very cautious about resource usage (and resource domination by a single client) anyway :)

flow, well, I'm coming from deletion of old items potentially being expensive.

flow, it's precisely about resource usage, just different resources 🙂

Holger, so in your scenario, a client that continiously creates pubsub items, would eventually run in which limit?

Right now we delete the oldest item(s) when a new one is published.

volume size of the volume the database is stored on?

That can be an expensive operation if you have millions of items.

(Which is why we don't support this with MAM, for example.)

doesn't have postgres some vacuum mechanism for that? ;)

Just the query to look up the oldest item(s) is too expensive.

Stuff them in buckets, sized after about the number of items you expect to delete each time, something something

But I'm not sure this matters. I mean I'm highly interested in implementation suggestions for my problem at hand. But I do think we shouldn't enforce having some limit ("just choose a large one!") protocol-wise, either way.

I am asking because I wonder if your "unlimited" is not really "very high and I can't provide the exact limit but there is one, I guarantee" limited ✏

There's no fixed limit, no.

What we're saying is that there is, when you run out of disk space :)

The server might clean up evey now and then, in my specific case.

so it's like max="best-effort"

and this is what "max" is for

It's just the client saying it wants no limit, so it will take the highest limit you have to offer (including no limit)

Let's not turn in circles 🙂

I'd also don't like to turn in circles ✏

Either way I do believe that the spec should clarify these things. Esp. whether the server is actually _supposed_ to return `max` if the client specified `max`.

In which case there's no way for the client to discover the limit right now.

definitely ✏

Whereas it means the limit might change.

(Well I guess it might always change anyway.)

Anyway, BBL.

And sneaking in some XEP-0122 element doesn't solve this?

What if: if there is a known max limit, let the pubsub service return it. if there is no known max limit, let the pubsub service return "max"

I'd push for the obvious solution of the server responding with the configuration option that the client set...

> I'd push for the obvious solution of the server responding with the configuration option that the client set... +1

The problem is that the max limit may change (e.g. because it's set by the operator, or based on available storage capacity), so the returned value is not very useful to the client anyway

right

"Obvious" 🙂

What is far more important is a way to configure the behaviour that occurs when the limit is reached

We don't have that, and the "delete oldest" behaviour is stupid for many applications

so another max_items enum value, but please do not call it "unlimited", seems sensible

I think the only compelling reason to return the current int equivalent rather than ‘max’ is that ‘max’ will break compliant implementations - and I do believe that’s a compelling reason.

Also we need to fix that ejabberd and Prosody disagree about the definition of "oldest" :)

But returning ‘max’ (with some additional way to discover what that means) is the conceptually right thing, I think.

Sure

MattJ, you mean we need to fix the specification to be clear and unambiguous when it comes to this ✏

Both are spec issues

hello

👋️

lovetox: sure thing; no rush though, if it's like 5 minutes before that's fine.

(sorry, I never remember how to PM on here)

Reminder to all that the Office Hours got pushed back an hour today, so show up in ~45 minutes and we'll get started! Today's is a demo of the new Gajim 1.4 UI and features! https://socialcoop.meet.coop/sam-pku-dud-niv

Kev, poooooooooooooooooooooooooooke https://github.com/xsf/xeps/pull/1047

Needs more XEP-0132

Oh man missed this one again! Pretty excited to see the PRs for Gajim flowing in though.