-
Holger
I'd like to support PubSub nodes with an unlimited number of items. AFAICS there's no `max_items=unlimited` thing. We now have `max_items=max`, but I think the meaning is a little different. (Plus I'm unsure whether there's an agreement on whether `max` is going to stay that way, now?)
-
Zash
And so it begins
-
Kev
max was as many as the deployment will allow, wasn’t it?
-
Holger
So my idea of how to proceed is: I'll submit a PR that adds support for `max_items=unlimited` to 0060, everyone's thrilled of the idea, it's merged this afternoon! That's how things work, right?
-
Holger
Kev, yeah.
-
Kev
Which, if a deployment allows unlimited items, would mean unlimited.
-
Kev
And if the deployment doesn’t allow unlimited would mean as close to unlimited as you can get.
-
Holger
So I guess the _idea_ was something like "client configures max_items=max" and sees the resulting limit when querying the config from the server.
-
Holger
Where "the resulting limit" is an integer.
-
MattJ
Not necessarily (IMHO)
-
Holger
Or is it fine for the server to return `max_items=max`?
-
Zash
IIRC yes
-
Zash
Only the server would know the true limit
-
Holger
Might be two questions: (1) is it fine as per the wording, (2) is it fine as per the author's intention.
-
MattJ
Given that that value may change over time (based on configuration, implementation default, etc.)
-
Zash
Clients would just see 'max'
-
Holger
Ok. Works for me. Kinda even better than my idea of how to proceed 🙂
-
Holger
So people were fine with the `max` thing, or at least gave up on going `-1` or something instead?
-
Holger
I.e. if I add support, there's a slight chance it's still in line with 0060 next month? 🙂
-
MattJ
There's a significant chance
-
Holger
\o/
-
Zash
Where's that PR?
-
MattJ
I think we decided it's probably easiest at this point to add a new type for this field
-
MattJ
(and similar fields in other XEPs)
-
Holger
The `max` thing is already in 0060.
-
Zash
🙁
-
MattJ
Holger, https://github.com/xsf/xeps/pull/988
-
Holger
I just was unsure whether it (a) can be (ab)used for my use case and if so, (b) whether it's there to stay.
-
Holger
Ah.
-
Holger
MattJ, thanks.
-
MattJ
"unlimited" will throw a spanner in the works
-
Holger
Heh.
-
Holger
I'll shut up then 🙂
-
Holger
And work with `max`.
-
MattJ
I think in my mind "max" was "unlimited" (or as high as is possible)
-
MattJ
which is why it doesn't make sense to translate it back to a number
-
Holger
Hrm.
-
MattJ
It's unfortunate that the client can't then know whether it's actually unlimited
-
Holger
Strictly speaking I would still think it does make sense.
-
Holger
Right.
-
Kev
It both does and doesn’t.
-
MattJ
You're right and wrong
-
Kev
I agree with 50% of that :)
-
Kev
A disco feature for max-means-unlimited might make sense, I guess. Or some way of making the max discoverable (which I don’t think it currently is, but might misremember)
-
Holger
I would still think the proper solution is to have both `unlimited` and `max`. They mean different things. Having max-means-$whatever in disco instead sounds weirdo to me.
-
MattJ
Having both seems weird
-
Kev
The two reasons that come to mind for it making sense to convert back to a number for display purposes (ideally while persisting ‘max’ internally) are that ‘max’ will break unsuspecting clients, and that it’s nice to know what the max is. The rason for not converting to an int is that you want the client to know it’s pinned at the max.
-
Holger
MattJ, why?
-
MattJ
If unlimited exists, why would anyone ever use max? Unlimited is higher than max, or at least implies there is no maximum
-
Kev
Or, why would anyone ever use unlimited, I guess. Because you probably want either unlimited or as close to it as you can get, which is what max gives you.
-
Holger
There's either a limit, expressed as a non-negative integer, or no limit, expressed as `unlimited`. The client won't know. So we have the special value `max` which means `unlimited` or, if not possible, the highest limit allowed.
-
MattJ
The whole "max" thing was added because clients wanted to be able to shove as much into the node as they could, without picking an arbitrary numer✎ -
Holger
Right.
-
Kev
*Unless* you want unlimited and anything less than that you want to error on.
-
MattJ
The whole "max" thing was added because clients wanted to be able to shove as much into the node as they could, without picking an arbitrary number ✏
-
Holger
Right.
-
MattJ
Which is exactly what you're after with "unlimited"
-
Holger
Yes.
-
Zash
And we couldn't have advertised the server limit because clients want it pinned to the max, even if it changes
-
Holger
In my specific use case that's actually what I'd want.
-
Holger
(Return an error if `unlimited` is not allowed.)
-
MattJ
I don't think it makes sense for a server that allows unlimited pubsub to also have an arbitrary "max" value
-
Kev
Holger: But in that case, the absense of a “max means unlimited” feature would work for that, no?
-
Holger
I can live without that error, but it would be the proper solution.
-
Holger
Kev, yes, I'm trying to argue for the proper solution being to have both `max` and `unlimited`. In which case max would of course not mean unlimited.
-
Zash
So we just shove more enum entries in that?
-
Zash
Why not `default` too‽
-
Holger
In my book, yes. enum entries are relatively cheap, no?
-
MattJ
Especially when mixed into the same data type as abitrary integers
-
Holger
> I don't think it makes sense for a server that allows unlimited pubsub to also have an arbitrary "max" value I don't quite understand.
-
Kev
> In which case max would of course not mean unlimited. No? I’d have expected when ‘unlimited’ was supported for ‘max’ to mean ‘unlimited’.
-
Holger
Kev, ah. Yes 🙂
-
MattJ
Holger, what's not to understand? What would "max" be if the server allows "unlimited"?
-
Holger
Seems I'm having a hard time to express myself.
-
MattJ
Client developers that pushed for "max" didn't want it because they wanted some arbitrary limit
-
Kev
Welcome to the Internet :)
-
Holger
If the server allows unlimited and the client requests `max` then the result is unlimited.
-
Zash
From the clients perspective, if they configure with limit=unlimited, then if it's supported, everything is ok, but if it's not, then they get a proper error.
-
MattJ
They pushed for it because they wanted no limit, but aren't necessarily going to get that
-
Daniel
Max was certainly intended to mean unlimited (from a client's perspective)
-
MattJ
^
-
Zash
Eugh
-
Kev
> Max was certainly intended to mean unlimited (from a client's perspective) With the realisation that some arbitrarily high limit might exist, because unlimited storage hasn’t been invented yet, except in /dev/null, right? :)
-
Holger
Daniel, my assumption was, for the case where the server allows 1000, `max` means 1000.
-
MattJ
Holger, that's correct. But such a server wouldn't also allow unlimited entries. Would it?
-
MattJ
If it did, then "max" == unlimited
-
Kev
What would probably be neatest would be for a server to return an annotation when a client configures the node to ‘max’ saying what the effective value of ‘max’ is, with ‘unlimited’ being a possible value :)
-
Kev
Such that a Holger client can abort because 9,223,372,036,854,775,807 isn’t high enough, but everyone else can not care :)
-
Holger
MattJ, I don't get it 🙂 If I configure my server to only allow 1000 items for a given node, then this server won't at the same time allow an unlimited number of items for that node, no.
-
Holger
But you obviously meant something else 🙂
-
Daniel
My wishful thinking was that what ever internal limits the server might have it won't matter in practice because they are hight enough
-
Daniel
So max would be a de facto unlimited
-
MattJ
Holger, no, that's exactly what I meant
-
MattJ
Just pretend `max` means unlimited and carry on :)
-
Holger
So if you guys are telling me to just use a super-large integer in place of `unlimited` because it has the same effect in practice then I at least get the idea. But no I'm not going to implement `if limit < 9,223,372,036,854,775,807 then delete_old_items()` 🙂
-
Holger
MattJ, yes yes, works for me.
-
MattJ
Are you talking about the client or server perspective here?
-
Holger
Server.
-
MattJ
I'm not sure why the server needs to care. Either you have a limit, or you don't
-
Holger
I'd like to allow clients to configure nodes without limit.
-
MattJ
If you don't, then max really does mean unlimited, not some arbitrary value
-
Zash
limit=max → high as it goes plz limit=unlimited → give me unlimited, or give me ERROR
-
Zash
Hm
-
Zash
Wouldn't adding entries to this enum require yetanotherxeprevision?
-
Holger
Yup. I totally get how this solution works in practice. Assuming it's fine to return `max_items=max` when the client asks for the node config.
-
MattJ
Which would error on most servers, because unrestricted resource consumption is unrealistic for most deployments :)
-
Zash
Aaaaaaaaah
-
MattJ
Holger, it is
-
Holger
I'll insist that strictly speaking, it is hacky and wrong, but it works for me 🙂
-
MattJ
It would be wrong to not return "max" in the node config
-
MattJ
Whether you have a limit or not
-
Holger
Wait, you're saying I should return `max` even in the case where there _is_ a limit?
-
MattJ
Yes, because that's what the configuration is set to, "max"
-
Holger
So for a node with max_items=1000. If the client configured `max_items=1000` I should return `1000`, if the client said `max_items=max` I should return `max`?
-
Kev
Did we bump the xep60 config namespace for this? Because if not, returning max to an unsuspecting client under the previous namespace that only supported ints would be wrong :D
-
MattJ
Kev, sure, it broke Prosody assumptions about it being an integer too *shrug* :)
-
MattJ
Why do clients get a namespace bump but servers don't?
-
Kev
Becuase the server would reject max as being invalid until it supported max?
-
flow
Holger, IIRC 'max' should only be send by the configuring entity and the real integer value is returned by the pubsub service
-
flow
but I could be wrong
-
Kev
(Although I note that there are definitely deployments that are doing checking against schemas (not necessarily XEP schemas), and so any changing of protocol like this without a bump breaks them, and that’s bad)
-
flow
and I woudldn't be suprised if this is not clearly stated in the XEP
-
Holger
flow, that's what I assumed, and then everyone told me I was wrong 🙂
-
Kev
(Checking about schemas on the way through a routing entity, that is)
-
Holger
Right, it's not clearly stated.
-
Kev
Holger: I think only Matt said that’s wrong :)
-
Zash
And why can't we do nice things like advertise the the types in the config from?✎ -
Zash
And why can't we do nice things like advertise the the types in the config form? ✏
-
flow
pep tried to get the types in data forms validation, but that processed stalled
-
MattJ
There are no schemas, since it's a data form
-
MattJ
and it's text-single
-
Holger
Kev, ok, and admittedly he only said half-wrong half-right, but in the end, wrong, or something 🙂
-
flow
Holger, I am curious: why do you want unlimited? as Kev pointed out, the inifnite tape of a turing machine is a theoretical construct ;)
-
Zash
MattJ, dataforms validation is sorta like schema tho?
-
MattJ
Zash, yes, but not used by XEP-0060
-
Zash
Nice things, can not have.
-
flow
and as server dev, I'd expect you to be very cautious about resource usage (and resource domination by a single client) anyway :)
-
Holger
flow, well, I'm coming from deletion of old items potentially being expensive.
-
Holger
flow, it's precisely about resource usage, just different resources 🙂
-
flow
Holger, so in your scenario, a client that continiously creates pubsub items, would eventually run in which limit?
-
Holger
Right now we delete the oldest item(s) when a new one is published.
-
flow
volume size of the volume the database is stored on?
-
Holger
That can be an expensive operation if you have millions of items.
-
Holger
(Which is why we don't support this with MAM, for example.)
-
flow
doesn't have postgres some vacuum mechanism for that? ;)
-
Holger
Just the query to look up the oldest item(s) is too expensive.
-
Zash
Stuff them in buckets, sized after about the number of items you expect to delete each time, something something
-
Holger
But I'm not sure this matters. I mean I'm highly interested in implementation suggestions for my problem at hand. But I do think we shouldn't enforce having some limit ("just choose a large one!") protocol-wise, either way.
-
flow
I am aksing because I wonder if your "unlimited" is not really "very high and I can't provide the exact limit but there is one, I guarantee" limited✎ -
flow
I am asking because I wonder if your "unlimited" is not really "very high and I can't provide the exact limit but there is one, I guarantee" limited ✏
-
Holger
There's no fixed limit, no.
-
MattJ
What we're saying is that there is, when you run out of disk space :)
-
Holger
The server might clean up evey now and then, in my specific case.
-
flow
so it's like max="best-effort"
-
MattJ
and this is what "max" is for
-
MattJ
It's just the client saying it wants no limit, so it will take the highest limit you have to offer (including no limit)
-
Holger
Let's not turn in circles 🙂
-
flow
I'd also don't like to turn in circle✎ -
flow
I'd also don't like to turn in circles ✏
-
Holger
Either way I do believe that the spec should clarify these things. Esp. whether the server is actually _supposed_ to return `max` if the client specified `max`.
-
flow
definetly✎ -
Holger
In which case there's no way for the client to discover the limit right now.
-
flow
definitely ✏
-
Holger
Whereas it means the limit might change.
-
Holger
(Well I guess it might always change anyway.)
-
Holger
Anyway, BBL.
-
Zash
And sneaking in some XEP-0122 element doesn't solve this?
-
flow
What if: if there is a known max limit, let the pubsub service return it. if there is no known max limit, let the pubsub service return "max"
-
MattJ
I'd push for the obvious solution of the server responding with the configuration option that the client set...
-
Daniel
> I'd push for the obvious solution of the server responding with the configuration option that the client set... +1
-
MattJ
The problem is that the max limit may change (e.g. because it's set by the operator, or based on available storage capacity), so the returned value is not very useful to the client anyway
-
flow
right
-
Holger
"Obvious" 🙂
-
MattJ
What is far more important is a way to configure the behaviour that occurs when the limit is reached
-
MattJ
We don't have that, and the "delete oldest" behaviour is stupid for many applications
-
flow
so another max_items enum value, but please do not call it "unlimited", seems sensible
-
Kev
I think the only compelling reason to return the current int equivalent rather than ‘max’ is that ‘max’ will break compliant implementations - and I do believe that’s a compelling reason.
-
MattJ
Also we need to fix that ejabberd and Prosody disagree about the definition of "oldest" :)
-
Kev
But returning ‘max’ (with some additional way to discover what that means) is the conceptually right thing, I think.
-
flow
MattJ, you mean we need to fix the specification but be clear and unambiguous when it comes to this✎ -
MattJ
Sure
-
flow
MattJ, you mean we need to fix the specification to be clear and unambiguous when it comes to this ✏
-
MattJ
Both are spec issues
-
support@mpro.la
hello
-
Zash
👋️
-
Sam
lovetox: sure thing; no rush though, if it's like 5 minutes before that's fine.
-
Sam
(sorry, I never remember how to PM on here)
-
Sam
Reminder to all that the Office Hours got pushed back an hour today, so show up in ~45 minutes and we'll get started! Today's is a demo of the new Gajim 1.4 UI and features! https://socialcoop.meet.coop/sam-pku-dud-niv
-
jonas’
Kev, poooooooooooooooooooooooooooke https://github.com/xsf/xeps/pull/1047
-
Zash
Needs more XEP-0132
-
deuill
Oh man missed this one again! Pretty excited to see the PRs for Gajim flowing in though.