HolgerI'd like to support PubSub nodes with an unlimited number of items. AFAICS there's no `max_items=unlimited` thing. We now have `max_items=max`, but I think the meaning is a little different. (Plus I'm unsure whether there's an agreement on whether `max` is going to stay that way, now?)
ZashAnd so it begins
Kevmax was as many as the deployment will allow, wasn’t it?
HolgerSo my idea of how to proceed is: I'll submit a PR that adds support for `max_items=unlimited` to 0060, everyone's thrilled of the idea, it's merged this afternoon! That's how things work, right?
KevWhich, if a deployment allows unlimited items, would mean unlimited.
KevAnd if the deployment doesn’t allow unlimited would mean as close to unlimited as you can get.
HolgerSo I guess the _idea_ was something like "client configures max_items=max" and sees the resulting limit when querying the config from the server.
HolgerWhere "the resulting limit" is an integer.
MattJNot necessarily (IMHO)
HolgerOr is it fine for the server to return `max_items=max`?
ZashOnly the server would know the true limit
HolgerMight be two questions: (1) is it fine as per the wording, (2) is it fine as per the author's intention.
MattJGiven that that value may change over time (based on configuration, implementation default, etc.)
ZashClients would just see 'max'
HolgerOk. Works for me. Kinda even better than my idea of how to proceed 🙂
HolgerSo people were fine with the `max` thing, or at least gave up on going `-1` or something instead?
HolgerI.e. if I add support, there's a slight chance it's still in line with 0060 next month? 🙂
MattJThere's a significant chance
ZashWhere's that PR?
MattJI think we decided it's probably easiest at this point to add a new type for this field
MattJ(and similar fields in other XEPs)
HolgerThe `max` thing is already in 0060.
HolgerI just was unsure whether it (a) can be (ab)used for my use case and if so, (b) whether it's there to stay.
MattJ"unlimited" will throw a spanner in the works
HolgerI'll shut up then 🙂
HolgerAnd work with `max`.
MattJI think in my mind "max" was "unlimited" (or as high as is possible)
MattJwhich is why it doesn't make sense to translate it back to a number
MattJIt's unfortunate that the client can't then know whether it's actually unlimited
HolgerStrictly speaking I would still think it does make sense.
KevIt both does and doesn’t.
MattJYou're right and wrong
KevI agree with 50% of that :)
KevA disco feature for max-means-unlimited might make sense, I guess. Or some way of making the max discoverable (which I don’t think it currently is, but might misremember)
HolgerI would still think the proper solution is to have both `unlimited` and `max`. They mean different things. Having max-means-$whatever in disco instead sounds weirdo to me.
MattJHaving both seems weird
KevThe two reasons that come to mind for it making sense to convert back to a number for display purposes (ideally while persisting ‘max’ internally) are that ‘max’ will break unsuspecting clients, and that it’s nice to know what the max is.
The rason for not converting to an int is that you want the client to know it’s pinned at the max.
MattJIf unlimited exists, why would anyone ever use max? Unlimited is higher than max, or at least implies there is no maximum
KevOr, why would anyone ever use unlimited, I guess. Because you probably want either unlimited or as close to it as you can get, which is what max gives you.
HolgerThere's either a limit, expressed as a non-negative integer, or no limit, expressed as `unlimited`. The client won't know. So we have the special value `max` which means `unlimited` or, if not possible, the highest limit allowed.
MattJThe whole "max" thing was added because clients wanted to be able to shove as much into the node as they could, without picking an arbitrary numer
Kev*Unless* you want unlimited and anything less than that you want to error on.
MattJThe whole "max" thing was added because clients wanted to be able to shove as much into the node as they could, without picking an arbitrary number
MattJWhich is exactly what you're after with "unlimited"
ZashAnd we couldn't have advertised the server limit because clients want it pinned to the max, even if it changes
HolgerIn my specific use case that's actually what I'd want.
Holger(Return an error if `unlimited` is not allowed.)
MattJI don't think it makes sense for a server that allows unlimited pubsub to also have an arbitrary "max" value
KevHolger: But in that case, the absense of a “max means unlimited” feature would work for that, no?
HolgerI can live without that error, but it would be the proper solution.
HolgerKev, yes, I'm trying to argue for the proper solution being to have both `max` and `unlimited`. In which case max would of course not mean unlimited.
ZashSo we just shove more enum entries in that?
ZashWhy not `default` too‽
HolgerIn my book, yes. enum entries are relatively cheap, no?
MattJEspecially when mixed into the same data type as abitrary integers
Holger> I don't think it makes sense for a server that allows unlimited pubsub to also have an arbitrary "max" value
I don't quite understand.
Kev> In which case max would of course not mean unlimited.
No? I’d have expected when ‘unlimited’ was supported for ‘max’ to mean ‘unlimited’.
HolgerKev, ah. Yes 🙂
MattJHolger, what's not to understand? What would "max" be if the server allows "unlimited"?
HolgerSeems I'm having a hard time to express myself.
MattJClient developers that pushed for "max" didn't want it because they wanted some arbitrary limit
KevWelcome to the Internet :)
HolgerIf the server allows unlimited and the client requests `max` then the result is unlimited.
ZashFrom the clients perspective, if they configure with limit=unlimited, then if it's supported, everything is ok, but if it's not, then they get a proper error.
MattJThey pushed for it because they wanted no limit, but aren't necessarily going to get that
DanielMax was certainly intended to mean unlimited (from a client's perspective)
Kev> Max was certainly intended to mean unlimited (from a client's perspective)
With the realisation that some arbitrarily high limit might exist, because unlimited storage hasn’t been invented yet, except in /dev/null, right? :)
HolgerDaniel, my assumption was, for the case where the server allows 1000, `max` means 1000.
MattJHolger, that's correct. But such a server wouldn't also allow unlimited entries. Would it?
MattJIf it did, then "max" == unlimited
KevWhat would probably be neatest would be for a server to return an annotation when a client configures the node to ‘max’ saying what the effective value of ‘max’ is, with ‘unlimited’ being a possible value :)
KevSuch that a Holger client can abort because 9,223,372,036,854,775,807 isn’t high enough, but everyone else can not care :)
HolgerMattJ, I don't get it 🙂 If I configure my server to only allow 1000 items for a given node, then this server won't at the same time allow an unlimited number of items for that node, no.
HolgerBut you obviously meant something else 🙂
DanielMy wishful thinking was that what ever internal limits the server might have it won't matter in practice because they are hight enough
DanielSo max would be a de facto unlimited
MattJHolger, no, that's exactly what I meant
MattJJust pretend `max` means unlimited and carry on :)
HolgerSo if you guys are telling me to just use a super-large integer in place of `unlimited` because it has the same effect in practice then I at least get the idea. But no I'm not going to implement `if limit < 9,223,372,036,854,775,807 then delete_old_items()` 🙂
HolgerMattJ, yes yes, works for me.
MattJAre you talking about the client or server perspective here?
MattJI'm not sure why the server needs to care. Either you have a limit, or you don't
HolgerI'd like to allow clients to configure nodes without limit.
MattJIf you don't, then max really does mean unlimited, not some arbitrary value
Zashlimit=max → high as it goes plz
limit=unlimited → give me unlimited, or give me ERROR
ZashWouldn't adding entries to this enum require yetanotherxeprevision?
HolgerYup. I totally get how this solution works in practice. Assuming it's fine to return `max_items=max` when the client asks for the node config.
MattJWhich would error on most servers, because unrestricted resource consumption is unrealistic for most deployments :)
MattJHolger, it is
HolgerI'll insist that strictly speaking, it is hacky and wrong, but it works for me 🙂
MattJIt would be wrong to not return "max" in the node config
MattJWhether you have a limit or not
HolgerWait, you're saying I should return `max` even in the case where there _is_ a limit?
MattJYes, because that's what the configuration is set to, "max"
HolgerSo for a node with max_items=1000. If the client configured `max_items=1000` I should return `1000`, if the client said `max_items=max` I should return `max`?
KevDid we bump the xep60 config namespace for this? Because if not, returning max to an unsuspecting client under the previous namespace that only supported ints would be wrong :D
MattJKev, sure, it broke Prosody assumptions about it being an integer too *shrug* :)
MattJWhy do clients get a namespace bump but servers don't?
KevBecuase the server would reject max as being invalid until it supported max?
flowHolger, IIRC 'max' should only be send by the configuring entity and the real integer value is returned by the pubsub service
flowbut I could be wrong
Kev(Although I note that there are definitely deployments that are doing checking against schemas (not necessarily XEP schemas), and so any changing of protocol like this without a bump breaks them, and that’s bad)
flowand I woudldn't be suprised if this is not clearly stated in the XEP
Holgerflow, that's what I assumed, and then everyone told me I was wrong 🙂
Kev(Checking about schemas on the way through a routing entity, that is)
HolgerRight, it's not clearly stated.
KevHolger: I think only Matt said that’s wrong :)
ZashAnd why can't we do nice things like advertise the the types in the config from?
ZashAnd why can't we do nice things like advertise the the types in the config form?
flowpep tried to get the types in data forms validation, but that processed stalled
MattJThere are no schemas, since it's a data form
MattJand it's text-single
HolgerKev, ok, and admittedly he only said half-wrong half-right, but in the end, wrong, or something 🙂
flowHolger, I am curious: why do you want unlimited? as Kev pointed out, the inifnite tape of a turing machine is a theoretical construct ;)
ZashMattJ, dataforms validation is sorta like schema tho?
MattJZash, yes, but not used by XEP-0060
ZashNice things, can not have.
flowand as server dev, I'd expect you to be very cautious about resource usage (and resource domination by a single client) anyway :)
Holgerflow, well, I'm coming from deletion of old items potentially being expensive.
Holgerflow, it's precisely about resource usage, just different resources 🙂
flowHolger, so in your scenario, a client that continiously creates pubsub items, would eventually run in which limit?
HolgerRight now we delete the oldest item(s) when a new one is published.
flowvolume size of the volume the database is stored on?
HolgerThat can be an expensive operation if you have millions of items.
Holger(Which is why we don't support this with MAM, for example.)
flowdoesn't have postgres some vacuum mechanism for that? ;)
HolgerJust the query to look up the oldest item(s) is too expensive.
ZashStuff them in buckets, sized after about the number of items you expect to delete each time, something something
HolgerBut I'm not sure this matters. I mean I'm highly interested in implementation suggestions for my problem at hand. But I do think we shouldn't enforce having some limit ("just choose a large one!") protocol-wise, either way.
flowI am aksing because I wonder if your "unlimited" is not really "very high and I can't provide the exact limit but there is one, I guarantee" limited
flowI am asking because I wonder if your "unlimited" is not really "very high and I can't provide the exact limit but there is one, I guarantee" limited
HolgerThere's no fixed limit, no.
MattJWhat we're saying is that there is, when you run out of disk space :)
HolgerThe server might clean up evey now and then, in my specific case.
flowso it's like max="best-effort"
MattJand this is what "max" is for
MattJIt's just the client saying it wants no limit, so it will take the highest limit you have to offer (including no limit)
HolgerLet's not turn in circles 🙂
flowI'd also don't like to turn in circle
flowI'd also don't like to turn in circles
HolgerEither way I do believe that the spec should clarify these things. Esp. whether the server is actually _supposed_ to return `max` if the client specified `max`.
HolgerIn which case there's no way for the client to discover the limit right now.
HolgerWhereas it means the limit might change.
Holger(Well I guess it might always change anyway.)
ZashAnd sneaking in some XEP-0122 element doesn't solve this?
flowWhat if: if there is a known max limit, let the pubsub service return it. if there is no known max limit, let the pubsub service return "max"
MattJI'd push for the obvious solution of the server responding with the configuration option that the client set...
Daniel> I'd push for the obvious solution of the server responding with the configuration option that the client set...
MattJThe problem is that the max limit may change (e.g. because it's set by the operator, or based on available storage capacity), so the returned value is not very useful to the client anyway
MattJWhat is far more important is a way to configure the behaviour that occurs when the limit is reached
MattJWe don't have that, and the "delete oldest" behaviour is stupid for many applications
flowso another max_items enum value, but please do not call it "unlimited", seems sensible
KevI think the only compelling reason to return the current int equivalent rather than ‘max’ is that ‘max’ will break compliant implementations - and I do believe that’s a compelling reason.
MattJAlso we need to fix that ejabberd and Prosody disagree about the definition of "oldest" :)
KevBut returning ‘max’ (with some additional way to discover what that means) is the conceptually right thing, I think.
flowMattJ, you mean we need to fix the specification but be clear and unambiguous when it comes to this
flowMattJ, you mean we need to fix the specification to be clear and unambiguous when it comes to this
MattJBoth are spec issues
Samlovetox: sure thing; no rush though, if it's like 5 minutes before that's fine.
Sam(sorry, I never remember how to PM on here)
Steve Killehas left
Steve Killehas joined
SamReminder to all that the Office Hours got pushed back an hour today, so show up in ~45 minutes and we'll get started! Today's is a demo of the new Gajim 1.4 UI and features! https://socialcoop.meet.coop/sam-pku-dud-niv