XSF Discussion - 2020-08-14

Hello Kev, for the Tweets for the Newsletter I would like to ask for access to the Twitter account. And maybe Mastadon aswell, if someone has the access, too. jcbrand, Seve:

What are sensible max size restrictions on pubsub items these days?

Size of the items?

Guus: I suppose you want to keep https://xmpp.org/extensions/xep-0060.html#subscriber-retrieve-returnall under 10k or whatever stanza size limit you may have

So, stanza size limit / max items (per rsm page, if supported) - some margins

it's a pitty that xep60 doesn't use the same scheme as MAM when returning multiple items

Let's invent time travel and send that idea back to when '60 was designed

Wasn't PubSub MAM a thing we discussed at Summit before?

PubSub MAM is a thing, we deploy it and it's in the XEP, no?

Unless I'm going mad.

I am probably not very knowledgeable on pubsub, but what exactly is the benefit of pubsub mam?

You can get the history of items on a node.

given that it appears that there is pubsub+rsm already

Xep60 gives you the 'current state', MAM gives you the history.

the history of items means that I can access previous revisions of an item with id 'A'?

As an analogy, xep60 gives you the filesystem, 313 gives you the backups.

Yes, it would show you the history of things published to the node.

i'd like to point out that I always found that motivation for PubSub+MAM missing in xep313

mam introduction currently states "or to view old items in a pubsub node."

and I always wondered "can't I read old items already with plain xep60?"

and i guess xep-pubsub-mam does not exist (yet)?

I thought I wrote pubsub into 313.

you did, but it was removed early this year

Oh.

That seems...not good.

it was removed a week or two back

it’s a separate XEP (not yet)

because I forgot to do things editor-wise

Version 0.7.0 (2020-03-20)

I was reminded by reading that paragraph

0.7.0 was published a few weeks back

https://github.com/xsf/xeps/pull/921 this one I need to process, will soon

IIRC the mam pubsub part was not that substantial, e.g. it still wasn't clear to me why and when I would want to use it, and what it can do for me (but maybe that's just me)

simply stating that you can access previous item revisions with it may be a good first step

but eventually I think there is much more to it, like the retention policy of this "historical" information

Isn't that exactly the same as with IM MAM?

So it'd let you replay notifications, or somesuch?

That's one way of looking at it I guess.

But if you're in an environment where pubsub is containing data that's a little bit like a message, rather than just transient notifications, the argument for MAM is the same as it is for IM, I think.

e.g. if you're doing FDP or something.

Notifications including retractions and other stuff

Found a 8 year old Issue from MattJ in Gajim, where he proposes to use stream management acks instead of pings to detect a timeout

whats the opinion about this today?

lovetox, timeout or liveness detection?

the later

lovetox, "yes please"

lovetox: if you send multiple <r/> without data in between, you can't decide which of them a resulting <a/> is for

Ge0rG, doesn’t matter

that you received an <a/> is enough to know that you’re still live

jonas’: doesn't matter for liveness detection, but does matter for server latency measurement

that is true, but you can do that passively

(if you really wanted to)

Ge0rG, why would i send multiple </r> though?

sm <r/> <a/> has fewer bytes on the wire compared to xmpp' pinging the server

like without waiting for the <a>

but IMHO not worth it, still a nice microoptimization

I think I'm using actual pings because I encountered some limitation in the Smack API...

and fewer corner cases ("what if the server does not support ping?")

lovetox: e.g. when doing it from a periodic timer

Ge0rG, also, you can decide because of in-order processing :)

yes Ge0rG but i would always wait for the <a> or? before issuing a new one?

lovetox, waiting for "the" <a/> is tricky

jonas’: but what if they get reordered or one is lost?!

Ge0rG, tcp?!

the main problem tho for corellation is: > Either party MAY send an <a/> element at any time (e.g., after it has received a certain number of stanzas, or after a certain period of time), even if it has not received an <r/> element from the other party.

jonas’: remember that fortigate TLS 1.3 issue I talked about recently?

please no

and second question is, do you think its a good idea to send <r> after a period of time rather than after a number of stanzas?

lovetox, I see no reason why you would request SM acks based on a period of time

lovetox, so I’d recommedn to send <r/> when you haven’t received anything from the server for more than N seconds

that you send <r/> specifically doesn’t matter, but sending *anything* which provokes a response from the server is good

<r/> lends itself because it’s cheap and has a beneficial side effect (updating the SM sync)

you request SM acks to drain your, potentially existing, unack'ed stanza queue

lovetox: it makes sense to send an <r/> after you sent out all the important stanzas that were queued, and then as jonas’ said after a longer period of idletime

then when you don’t receive *anything* (not specifically <a/>, anything from the server is fine) after another timeout, you consider the stream dead

2 seperate issues

this is much simpler than keeping track of pings and it’s also much better, because it can better deal with saturated high-latency links

lets move away from alive detection

but you were talking about liveness detection?

currently i request r after each stanza

that’s certainly a valid strategy

yes jonas’ and now i want to talk about the second issue

but unrelated to liveness detection

oh

but I hadn’t answered your first question yet :P ✏

:D, you said its a good idea

lovetox, I’d say for non-aliveness-detection purposes, do <r/> on each socket flush

Ge0rG, has some server latency issue i dont undestand and im not sure i want to care

I have a server latency issue?

jonas’, as i dont fliush the socket myself this is not a option

-ENOPARSE

i can ask for r after i get the callback that my write was successful

which is probably the same

if you don't flush the socket, maybe Nagle will

but im not sure that is really a optimization

lovetox: it's the opposite of an optimization

but i would have to meassure that first

lovetox: ideally, you want the stanza(s) and the <r/> to go out in the same TCP frame

instead of wasting roundtrips

yeah i use a library that abstracts that for me, so i have no idea when and how much is in a tcp frame

I know this is a controversial opinion, but I consider Gajim to be a mobile client when I'm on 2G tethering, riding a train in the middle of German nowhere.

lovetox: there is also TLS in between ;)

and compression

so it's borderline impossible to find out.

lovetox, then maybe wait for 1ms after you wrote a stanza to the socket and send a <r/> after that

jonas’: but why?

yeah, jonas’, why?

lovetox, and reset the timer after each stanza

flow, Ge0rG, to avoid sending multiple <r/>s for no good reason?

I think I lost slighlty what this discussion is about

yeah why multiple r?

look i have a send_stanza method

simply send an <r/> after a few stanzas or when you hit a high watermark of unack'ed stanzas

and everytime this is called, i also call send_r

its very simple

there can never be multiple r for one stanza

its alway stanza, r, stanza, r, stanza, r

flow, so you send <r/> after N stanzas and the "last" batch of stanzas has N-1. You’re now left with N-1 unacked stanzas "forever".

for no good reason.

jonas’, well if the unacked stanzas are not important then this is fine

thats why my idea was, something like, every 3 seconds (of course only if there are unacked stanzas=

flow, but if they are? what makes a stanza "important"?

lovetox, please don’t send multiple <r/> for any unacked stanza

smack has predicates that decide when to issue an <r/> ✏

you’ll just pile on to an apparently already existing network congestion

trivial example is "messages with <body/>"

especially not as often as every 3 seconds.

jonas’: a stanza is important if it is not ephemeral and not going to time out if left in hibernation

jonas, of course only one r is in flight

please

lovetox, it’ll stay in-flight forever though

if the server doesn’t reply

or if the server takes long to reply

but it MUST reply

yes, but taht may take some time

yeah and?

e.g. 5 minutes on a crappy mobile link

in that time you have amassed 100 <r/>s

to which the server MUST reply

so not only have you congested your uplink, but also your downlink now

unneccesarily

i propose to send an r every 3 seconds only if there is no r in flight

so how would i amass 100 r

oh, you messed up a negation earlier then

maybe, although i dont see it

oh, I imagined an "if" in a sentence when there was none

sorry for that

lovetox, so, problem is, tracking <r/> and <a/> isn’t really possible

so how would you define "in-flight"?

every 3 seconds?!

Ge0rG, (of course only if there are unacked stanzas)

Ge0rG, + the in-flight this results in "do <r/> if there are unacked stanzas 3s after the last stanza has been sent" or something like that

that's an interesting trade-off.

which is kind of what I proposed earlier, just with a much higher delay, causing less battery efficiency

and with an unclear definition of "in-flight"

one could switch the dealy dynamically

How long do you wait after generating a stanza and before concluding that there are no more stanzas going to be sent

when i have unacked stanzas its 3s, afterwards its 20

or soemothing

lovetox, what do you send an <r/> for if you don’t have unacked stanzas? ✏

for alive detection ^^

back to issue one

lovetox, see what I wrote in regard of that one please then

maybe it's a good thing to just put all the stanzas that you want to send into a queue, then let the OS/event API know you have something to send, and then add a single <r> at the end of the queue

...on delivery

generally i want to send every data instantly

Ge0rG, a simpler version of that, requiring less cooperation from the OS, is to re-implement that TCP-cork-ish stuff in userspace: N ms after the last stanza has been put in the tx queue of the socket, add <r/>. reset the timer on each stanza.

to put it into a queue then wait some time, maybe more comes

seems not very fast

lovetox, the OS will do that for you anyways. and it’s ok, since the time is generally rather short.

jonas’: but then you are competing with TCP cork and end up sending the <r> in a separate TLS frame in a separate TCP frame

so you add something like 20x the overhead to <r/>

Ge0rG, only if your delay is longer than the tcp delay

which you can query from the OS

and if that’s too complex for you (I don’t blame you), then just send <r/> after each stanza

or just give it up

ok i keep sending r after each stanza

i dont send many stanzas anyway

Ge0rG, delay on linux is 200ms

lovetox, don’t implement IBB then :D

hehe

yeah

i already have

do you request an <r> for each ibb stanza?

:)

so it was YOU crashing my server? :D

how do <r/>s cause a crashing server?

do I want to know?

jonas’: j/k

you never know

you find the weirdest bugs

thanks! ✏

lovetox, https://docs.zombofant.net/aioxmpp/devel/api/internal/protocol.html#aioxmpp.protocol.XMLStream.deadtime_soft_limit here’s how liveness detection it works in aioxmpp. The docs at that place are written from a very internal perspective, hence user == the aioxmpp client class in this case. -> when the soft timeout expires, the stream sends <r/>. when the hard timeout expires, the stream is considered dead. timeouts reset when (any) data is received (it doesn’t even have to be a stanza).

the frontend to that is https://docs.zombofant.net/aioxmpp/devel/api/public/stream.html#timeouts-stream-aliveness-checks

("frontend")

thanks i look into it

if anything is unclear, feel free to ask. if anything seems stupid, please let me know, because currently I think this is the best anyone can do given networking realities.

(and what I particularly love about it is the simplicity, because it only uses two timers. no tracking of stanzas or whatnot)

(and the building blocks can easily be reused for MUC aliveness checking and stuff like that)

(because the concepts are really the same everywhere)

also relevant: https://docs.zombofant.net/aioxmpp/devel/api/internal/utils.html#aioxmpp.utils.AlivenessMonitor ✏

oh and this is also an amazing side-effect: "If the peer is pinging us in an interval which works for us (i.e. is less than the soft limit), we don’t need to ping the peer; no extra logic required."

yeah less timers the better

i hate timers

two timers, no tracking.

not sure what's the best place to ask (maybe jdev?) but here goes - is there somewhere a description of the current setup of xmpp.net? or it's sources? "about" (https://xmpp.net/about.php) only states that it's based on xmpppoke and it seems that https://github.com/stpeter/xmppdotnet is not exactly the same thing

Wojtek: https://github.com/xmpp-observatory

Thank you!