jdev - 2022-01-31

note that there is also message-based IBB, which appears to be not negotiated (maybe it should be)

flow, should it? IBB is always used within the context of another protocol, so it should be that protocol's job to figure it out (for instance, jingle)

I think so yes, mostly because I don't see much of reason for message based IBB (besides not having to implement IQ semantics, which appears to be a weak argument)

and being able to throttle IBB transfers by delaying the IQ response seems like a very important mechanism to have (not saying that each and every implementation has to use it)

I was thinking more about this yesterday and I think I'm just going to implement the error. Even if the other side doesn't understand it and terminates the connection, which isn't ideal, it's all I can do really. I still don't want to allow unrestricted buffer growth

From the clients perspective I'm not sure that throttling gets you much since there aren't negotiated timeouts, so if you hold the IQ that would put you over your data query it may just time out and you'll receive it again and again

What buffer are you talking about?

wait, clients re-send IQs which timed out?

I sure hope not

not by default anyway

I've not met one that does, that sounds like a terrible idea.

I dunno, maybe? Either way they either resend it or cancel the connection, a wait error at least indicates that they *should* resend it (even though they still might not if they don't support it), so same thing but slightly better

oops, I meant the buffer of received data

If for some reason it's not being read fast enough, I don't want that buffer to grow indefinitely, so at some point we limit it and I'm trying to figure out what to do in that case where it's maxed out and we can't receive more data until some gets read

how about the lib doesn't buffer at all, but offers only a callback instead? :)

Something Promise-ish? I imagine usually the data would go through the buffer and then into a file write or such. Returning the iq-response after writing might have sensible self-regulating effects I would think?

I thought about it, but I didn't like the semantics of that as much. I want to give the user stream semantics, not make them do it all themselves. Ie. I did an experiment with doing jingle and it was a *lot* easier if the underlying transports all had the same stream semantics

Writing the IQ-response after the data gets written somewhere else (or from our perspective after it gets read) is the other option, but I think I'm more worried about IQ timeouts and the unknown of whatever that means for the connection state

I wouldn't worry about timeouts.

Why's that? Wouldn't it cause problems?

timeouts are always wrong

so making them more or less wrong on the responder side doesn't matter ;)

Sure, but they exist and I feel like both sides have to agree what they mean if the stream is to continue

NN

Oops, I can never remember how to scroll in Mcabber despite doing it all the time.

The other problem is that if I hold the response, the other end times out waiting for it, and their response is to send the next packet I effectively end up just buffering it all anyways waiting for the data to be decoded and written

I dunno, maybe it doesn't matter until I have a practical use for this or someone else starts using it.

sending the next IBB data IQ if the response of previous times out doesn't appear the right thing to do™

That doesn't mean the other side won't do it; the spec specifically says you don't have to wait for responses

besides the case where the sender sends multiple IBB data IQs at once

Sam, not arguing that there could be implementations that behave that way, just saying that it's probably not sensible

Yah, totally agree; maybe it's worth clarifying the spec to say "this is how you say *back off*", or maybe that's just unnecessary complexity and I should just let the buffer grow

Let whatever larger protocol is using IBB deal with this.

unbounded buffer grow also doesn't sound very appealing

being able to delay the IBB IQ response is IMHO very sensible and sending entities should consider that

how weird would it be to throttle reading from your TCP connection?

that said, I believe many implementations are just fine without

Indeed (to both of those)

Zash, weird in a way that you would throttle everything that goes over that TCP connection

Throttling at the tcp level wouldn't necessarily fix the problem though, and would back up all the stanzas afterwards

Throttling in general wouldn't really fix it; you still might be reading slower than the throttled writes are coming in.

well throttling also could mean annoucing a window size of 0, so hopefully there would be no more incoming writes coming in

where "window size" here refers to a concept on the XMPP layer, not the TCP window size

in general, besides delaying the IBB IQ response, the IQ response could contain the remaining capacity of the buffer (or something similar)

of course, we then would lack a mechanism to annouce that the capacity changed from zero to non-zero ✏

Maybe I should just see what other implementations are doing (if anything) and do whatever that is

I wanted to look what Smack does, but sadly did not had the time yet

Interesting; when I get a "close" from the other side I flush any data, Smack appears to consider the entire connection already closed and doesn't do that

Not sure that it matters, I just assumed my side wasn't closed until the close response goes through

I believe (from memory) that the RFC tells you what to do when sending closes (and I *think* it tells you to wait for the other side's close, but would need to check)

flow: Smack appears to only buffer at max one block size; I'm not sure what it does if the buffer hasn't been cleared yet, block I guess, so maybe it's doing the IQ wait thing

there's some sort of separate data queue, I guess I should go see if that has a limit

(doesn't seem to)

So nevermind, maybe smack just has unbounded buffer growth

Sam: you can have stream semantics with a bounded buffer right? If it's full the write just blocks/writes nothing? That's how all stream APIs I can think of work anyway

moparisthebest: this is for reads, but we actually do block from the users perspective. Ie. if data comes in via an IQ and there's space in the buffer we write it to the buffer. If there's not enough space left, I'm trying to figure out what to do. From the users perspective if there's anything in the buffer we read from it. If there's nothing in the buffer we block until something gets written to the buffer then we read it.

ah yea, hairy situation

Sam, it appears Smack has an unbounded queue here, which is far from ideal. so yes, it is similar to an unbounded buffer

(technically the queue is not unbounded, but INT32_MAX bounded, but still)