-
Martin
@ping trashserver.net
-
Martin
@ping diebesban.de
-
Ellenor Malik
A.
-
Ellenor Malik
What would cause Prosody to hog CPU?
-
Martin
For me it was mam hitting 10000 messages in one archive and using fine storage.✎ -
Martin
For me it was mam hitting 10000 messages in one archive and using internal file storage. ✏
-
Link Mauve
For me, smacks going through many queued messages for a given user.
-
Martin
> What would cause Prosody to hog CPU? Something enlightening might be found in your logs.
-
Maranda
Martin: define "hog"
- Maranda has recently upped the archives cap to 23000.. Is still waiting to see the performance hit.
-
Maranda
@version myst.blue
-
Echo1
Maranda: myst.blue is running Metronome version 3.13.5 on Linux
-
Martin
Maranda: > Martin: define "hog" Why are you asking me?
-
Maranda
because you mentioned "flat files storage" in the same sentence
-
Martin
But the hig came from the citation so the citation is not the same sentence and you should ask Ellenor Malik✎ -
Martin
But the hog came from the citation so the citation is not the same sentence and you should ask Ellenor Malik ✏
-
Ellenor Malik
GRR
-
Maranda
Martin, should I? Ellenor, didn't mention MAM, didn't mention flat files, didn't mention "after reaching ten thousands of messages in an archive" which is what I just raise the CAP from.... :)
-
Maranda
I'm more interested in numbers from that
-
Martin
But she brought the term "hog" into the ring, so if you ask definitions for "hog" you should ask her.
-
Martin
For me it was: prosody going up to 100% and being unusable (clients disconnected and couldn't log in again) pasting debug log into prosody muc I was asked if I have a huge archive. By counting my users archive it turned out it was exactly 10000, so the limit for archive messages in flat file storage. After stopping prosody, migrating to xmlarchive and restarting it was working/usable again.
- Maranda is puzzles by *how actually big* was that _10000_
- Maranda is puzzled by *how actually big* was that _10000_
-
Martin
Afair it was something between 10 and 20 MB
-
Martin
du -sh /var/lib/prosody/mdosch%2ede/archive/ 21M /var/lib/prosody/mdosch%2ede/archive/ du -sh /var/lib/prosody/mdosch%2ede/archive/martin.list 12M /var/lib/prosody/mdosch%2ede/archive/martin.list
-
Maranda
-rw-r----- 1 metronome metronome 18M May 1 15:13 maranda.dat ---> I'll have to do some testing on cpu usage when Metronome loads/stores the archive into/from RAM
-
Martin
Don't know why but it seems when I hit the 10000 cap prosody freaked out, took 100% CPU and was not usable anymore.
-
Martin
Usually it idles around having almost no CPU impact.
-
jonas’
Martin, that sounds familiar
-
jonas’
verily
-
jonas’
cc @ Ge0rG ^
-
Maranda
well I think my current storage strategy with atomic writes will be fine until datastores stay within the underlying (os/)hardware IO performance range✎ -
Maranda
well I guess my current storage strategy with atomic writes will be fine until datastores stay within the underlying (os/)hardware IO performance range✎ ✏ -
Maranda
well I guess my current storage strategy with atomic read/writes will be fine until datastores stay within the underlying (os/)hardware IO performance range ✏
-
Ge0rG
Martin: that issue almost killed yax.im yesterday. Apparently internal storage is not fast at all.
-
Ge0rG
Martin: there will be a solution to this soon, I'm sure.
- Maranda suspects it's not about internal storage
-
Ge0rG
Maranda: it is. deleting one line from internal storage requires fully rewriting the huuuuuge serialized file
-
Martin
Ge0rG: I was hoping that xmlstorage might help already. Maybe the structured data is easier to process.
-
Maranda
Ge0rG, that's not a problem here tbh, and me thinks it's more about that ".list" file there
-
Maranda
and that also you shouldn't keep to rewrite one huge serialized file to disk everytime just one line is popped
-
MattJ
Ge0rG [18:49]: > Martin: there will be a solution to this soon, I'm sure. There is no real plan to fix it, it's been known that this does not perform well since before the release, which is why we tell people not to have large archives with internal storage
-
MattJ
It's possible that in a future release we will merge xmlarchive or a derivative
-
MattJ
And maybe we can tweak e.g. removing more than one item at a time
-
MattJ
But rewriting how the archive store works on the stable branch is not on the cards, and would be a large project
-
Ge0rG
MattJ: does this affect the stable branch? I've only encountered it after going to trunk
-
Ge0rG
and IIRC the 10k limit is the default, and it already makes prosody almost break down ;)
-
MattJ
I thought it was in 0.11.x but not certain
-
Ge0rG
MattJ: I'm sure I'd have run into it before then.
-
MattJ
And the default is not to persist MAM at all actually :)
-
MattJ
(I don't believe in shipping software that logs peoples' messages by default)
-
Ge0rG
I don't believe in services that don't do everything they can to deliver messages
-
MattJ
mod_offline