jdev - 2020-06-29

lovetox, I don’t see any issue here, so unlikely: https://github.com/horazont/muchopper/issues

Do any popular servers actually implement XEP-0368? If yes, could anyone give point to any such servers?

Жокир, https://compliance.conversations.im/ any suggested as "compliant" servers should do, at least for c2s

I don’t know of any s2s implementation except maybe https://github.com/surevine/Metre , which isn’t quite a server.

Looking at my server log, I'm noticing that I'm getting a lot of connection timeouts on s2s in bursts - presumably x minutes after a user that caused the federation to be set up sent its last presence update.

I wonder if it'd be good to introduce a small factor of randomness to the timeout interval, to avoid staggered behavior.

Guus, maybe controversal counter-question: that does sounds like an timeout enforced on the application layer. if so, then why would you have an application layer timeout for s2s connections and not simply let the tcp connection timeout

flow, save resources.

the tcp connection will also not time out ever

is it worth it?

because both peers can see each other (in this scenario)

file descriptors are limited and when you notice you’re running out of them, it’s too late

being a bit proactive about preserving them is generally a good idea

ok so kill idle connections based on the amount of available file descriptions, but not based on time

you don’t know the amount of available file descriptors

(or, to be precise, only as second criteria based on time)

you know the limit, but you don’t know how many are open in your process

you can estimate, but you can be wrong in the bad direction.

(or in both directions, depending on how you estimate)

ls /proc/$pid/fd/ | wc -l

?

that’s at least a rather expensive way to do it

but true, that works, on systems where procfs has that feature

why is it expensive?

flow I don't mind much either way - I'm just noticing that I get a lot of disconnects. I'm thinking Prosody does this? Openfire probably does so as well, though.

flow, that’s many syscalls

(at the very least, it's configurable)

I can’t see immediately in man 5 procfs whether /proc/$pid/fd is a linux or a posix thing

jonas’, I wouldn't be surpised if there is a more efficient way to get that number

I would

I don't mind much closing idle connections (although it does feel like premature optimization a bit.)

especially on linux

I think I looked into that already and found that it’s not possible

there’s surely a reason why sudo does a for i in 0..MAXFD do close($i); done

as Openfire is a multi-platform solution, depending on any platform specific thingy is going to be a pain.

unless Java exposes things, which I doubt.

Guus, UnixOperationSystemMXBean.getOpenFileDescriptorCount()

not sure if something like that also exist for other OS'es

*Unix*OperationSystemMXBean is likely going to fail on Windows? 🙂

but also: not worth the complexity, maybe?

so you may have to implement a fallback strategy for sure (like disconnection based on a timeout)

I don’t see any problem with a timeout here, to be honest

Guus, potentially, depends on your goals I'd say

everything else seems slightly overengineered

file descriptors may also just be one reason why you want to keep the number of open connections low

other reasons may include running behind a stateful firewall and wanting to conserve resources there

I was just suggesting to add a small random factor in the timeout delay, nothing more 😉

Guus, which is always a good idea

also, given how I see batches of s2s tear down only to be brought up again, I'm suspecting that the default timeout of (Prosody?) might be on the low end.

prosody doesn’t have a default timeout

oh, that's interesting

note that I didn't actually check what server software is used on those. I just assumed.

having had a closer look: might be ejabberd 🙂

https://sotecware.net/files/noindex/connections.png

:-)

looks like a 30 minute timeout

combined with an hourly cron job maybe?

I think there are two timeouts, one ~15min (the linear curve down) and one ~30min (which also looks randomized, because of the slight exp-y behaviour at the end)

and yes, this is the connection stats of search.jabber.network, and the spikes you see is the hourly scan :)

(maybe randomize your scan!)

now we only need to identify the implementations with the 15m and 30m (randomized) timeout

Guus, it’s already shuffled :)

and what is keeping the baseline of 1.5k connections

moar shuffling!

flow, compare the ratios with https://search.jabber.network/stats#software :)

assuming that many "unknowns" are in fact prosody MUCs, because prosody doesn’t report version on MUC by default IIRC

ahh, so it is probably prosody which keeps the connections

flow, very likely

but the amount of 15m and 30m timeout connections appears to be nearly equal

I experimented with loading mod_s2s_idle_timeout or whatsitcalled on s.j.n, but then I disabled it to reduce the codebase to the minimum for some unrelated testing

flow, I’ll have to dig deeper into it, it’s also possible that the two different falloff behaviours there are an artifact of how the scanner works ✏

i see

since there are two scanning components, and one finishes much quicker than the other; it’s possible that the quicker one is causing the additional tip of the initial spike, while the slower one is what causes the slow fall off at the end

since the quicker one also tends to touch more domains

oh yeah, that’s very plausible

that may also explain the exp falloff due to shuffling

if there’s really just a 15m or something timeout involved

jonas’, are you aware that 'German' appears twice in the room languages table?

yes

de-de vs. de

I need to normalize that

https://sotecware.net/files/noindex/connections-1h.png

https://sotecware.net/files/noindex/ingestion-1h.png

that seems to fit very well

(the "filled" part in the second graph is the fast component, the "line" part in the second graph is the slow component)

the fast component ends at 07:24, which is exactly when the initial spike drops in the first graph