XMPP Service Operators - 2020-05-01

@ping trashserver.net

@ping diebesban.de

A.

What would cause Prosody to hog CPU?

For me it was mam hitting 10000 messages in one archive and using internal file storage. ✏

For me, smacks going through many queued messages for a given user.

> What would cause Prosody to hog CPU? Something enlightening might be found in your logs.

Martin: define "hog"

@version myst.blue

Maranda: myst.blue is running Metronome version 3.13.5 on Linux

Maranda: > Martin: define "hog" Why are you asking me?

because you mentioned "flat files storage" in the same sentence

But the hog came from the citation so the citation is not the same sentence and you should ask Ellenor Malik ✏

GRR

Martin, should I? Ellenor, didn't mention MAM, didn't mention flat files, didn't mention "after reaching ten thousands of messages in an archive" which is what I just raise the CAP from.... :)

I'm more interested in numbers from that

But she brought the term "hog" into the ring, so if you ask definitions for "hog" you should ask her.

For me it was: prosody going up to 100% and being unusable (clients disconnected and couldn't log in again) pasting debug log into prosody muc I was asked if I have a huge archive. By counting my users archive it turned out it was exactly 10000, so the limit for archive messages in flat file storage. After stopping prosody, migrating to xmlarchive and restarting it was working/usable again.

✏

Afair it was something between 10 and 20 MB

du -sh /var/lib/prosody/mdosch%2ede/archive/ 21M /var/lib/prosody/mdosch%2ede/archive/ du -sh /var/lib/prosody/mdosch%2ede/archive/martin.list 12M /var/lib/prosody/mdosch%2ede/archive/martin.list

-rw-r----- 1 metronome metronome 18M May 1 15:13 maranda.dat ---> I'll have to do some testing on cpu usage when Metronome loads/stores the archive into/from RAM

Don't know why but it seems when I hit the 10000 cap prosody freaked out, took 100% CPU and was not usable anymore.

Usually it idles around having almost no CPU impact.

Martin, that sounds familiar

verily

cc @ Ge0rG ^

well I guess my current storage strategy with atomic read/writes will be fine until datastores stay within the underlying (os/)hardware IO performance range ✏

Martin: that issue almost killed yax.im yesterday. Apparently internal storage is not fast at all.

Martin: there will be a solution to this soon, I'm sure.

Maranda: it is. deleting one line from internal storage requires fully rewriting the huuuuuge serialized file

Ge0rG: I was hoping that xmlstorage might help already. Maybe the structured data is easier to process.

Ge0rG, that's not a problem here tbh, and me thinks it's more about that ".list" file there

and that also you shouldn't keep to rewrite one huge serialized file to disk everytime just one line is popped

Ge0rG [18:49]: > Martin: there will be a solution to this soon, I'm sure. There is no real plan to fix it, it's been known that this does not perform well since before the release, which is why we tell people not to have large archives with internal storage

It's possible that in a future release we will merge xmlarchive or a derivative

And maybe we can tweak e.g. removing more than one item at a time

But rewriting how the archive store works on the stable branch is not on the cards, and would be a large project

MattJ: does this affect the stable branch? I've only encountered it after going to trunk

and IIRC the 10k limit is the default, and it already makes prosody almost break down ;)

I thought it was in 0.11.x but not certain

MattJ: I'm sure I'd have run into it before then.

And the default is not to persist MAM at all actually :)

(I don't believe in shipping software that logs peoples' messages by default)

I don't believe in services that don't do everything they can to deliver messages

mod_offline