For me it was mam hitting 10000 messages in one archive and using fine storage.✎
Martin
For me it was mam hitting 10000 messages in one archive and using internal file storage. ✏
Link Mauve
For me, smacks going through many queued messages for a given user.
Martin
> What would cause Prosody to hog CPU?
Something enlightening might be found in your logs.
solhas left
solhas joined
ackerman1scotthas left
jayteeukhas left
jayteeukhas joined
ackerman1scotthas joined
cincohas left
cincohas joined
solhas left
solhas joined
rintinfinnhas joined
Ellenor Malikhas left
Ellenor Malikhas joined
jayteeukhas left
jayteeukhas joined
robertooohas left
perflysthas left
solhas left
solhas joined
rintinfinnhas left
rom1dephas left
rom1dephas joined
rintinfinnhas joined
jayteeukhas left
jayteeukhas joined
rintinfinnhas left
joerghas left
jayteeukhas left
jayteeukhas joined
joerghas joined
joerghas left
joerghas joined
ackerman1scotthas left
holgerhas left
rintinfinnhas joined
joerghas left
holgerhas joined
joerghas joined
sonnyhas joined
solhas left
solhas joined
rintinfinnhas left
rintinfinnhas joined
ackerman1scotthas joined
solhas left
solhas joined
robertooohas joined
rintinfinnhas left
Echo1has joined
nmhsmhas joined
holgerhas left
Maranda
Martin: define "hog"
holgerhas joined
edhelashas left
Marandahas recently upped the archives cap to 23000.. Is still waiting to see the performance hit.
ru_maniachas left
rintinfinnhas joined
solhas left
solhas joined
insanityhas joined
edhelashas joined
ru_maniachas joined
Maranda
@version myst.blue
Echo1
Maranda: myst.blue is running Metronome version 3.13.5 on Linux
ru_maniachas left
ru_maniachas joined
rintinfinnhas left
rintinfinnhas joined
404.cityhas joined
solhas left
solhas joined
nmhsmhas left
rintinfinnhas left
Martin
Maranda:
> Martin: define "hog"
Why are you asking me?
Maranda
because you mentioned "flat files storage" in the same sentence
Martin
But the hig came from the citation so the citation is not the same sentence and you should ask Ellenor Malik ✎
Martin
But the hog came from the citation so the citation is not the same sentence and you should ask Ellenor Malik ✏
Ellenor Malik
GRR
404.cityhas left
Maranda
Martin, should I? Ellenor, didn't mention MAM, didn't mention flat files, didn't mention "after reaching ten thousands of messages in an archive" which is what I just raise the CAP from.... :)
Maranda
I'm more interested in numbers from that
Martin
But she brought the term "hog" into the ring, so if you ask definitions for "hog" you should ask her.
Licaon_Kterhas left
Martin
For me it was: prosody going up to 100% and being unusable (clients disconnected and couldn't log in again) pasting debug log into prosody muc I was asked if I have a huge archive. By counting my users archive it turned out it was exactly 10000, so the limit for archive messages in flat file storage. After stopping prosody, migrating to xmlarchive and restarting it was working/usable again.
rintinfinnhas joined
Marandais puzzles by *how actually big* was that _10000_
du -sh /var/lib/prosody/mdosch%2ede/archive/
21M /var/lib/prosody/mdosch%2ede/archive/
du -sh /var/lib/prosody/mdosch%2ede/archive/martin.list
12M /var/lib/prosody/mdosch%2ede/archive/martin.list
Maranda
-rw-r----- 1 metronome metronome 18M May 1 15:13 maranda.dat ---> I'll have to do some testing on cpu usage when Metronome loads/stores the archive into/from RAM
solhas left
solhas joined
Martin
Don't know why but it seems when I hit the 10000 cap prosody freaked out, took 100% CPU and was not usable anymore.
Martin
Usually it idles around having almost no CPU impact.
perflysthas joined
ackerman1scotthas left
Neustradamushas joined
Neustradamushas left
jayteeukhas left
jayteeukhas joined
ackerman1scotthas joined
jonas’
Martin, that sounds familiar
jayteeukhas left
jonas’
verily
jonas’
cc @ Ge0rG ^
jayteeukhas joined
Neustradamushas joined
Neustradamushas left
ru_maniachas left
insanityhas left
ru_maniachas joined
Neustradamushas joined
Neustradamushas left
solhas left
solhas joined
insanityhas joined
Maranda
well I think my current storage strategy with atomic writes will be fine until datastores stay within the underlying (os/)hardware IO performance range✎
Maranda
well I guess my current storage strategy with atomic writes will be fine until datastores stay within the underlying (os/)hardware IO performance range✎✏
Maranda
well I guess my current storage strategy with atomic read/writes will be fine until datastores stay within the underlying (os/)hardware IO performance range ✏
insanityhas left
insanityhas joined
Licaon_Kterhas joined
solhas left
solhas joined
perflysthas left
solhas left
solhas joined
stpeterhas joined
stpeterhas left
jayteeukhas left
jayteeukhas joined
perflysthas joined
solhas left
solhas joined
jayteeukhas left
jayteeukhas joined
jayteeukhas left
jayteeukhas joined
perflysthas left
kusonekohas left
kusonekohas joined
kusonekohas left
kusonekohas joined
solhas left
solhas joined
kusonekohas left
kusonekohas joined
kusonekohas left
kusonekohas joined
solhas left
ijhas left
perflysthas joined
jayteeukhas left
jayteeukhas joined
ijhas joined
solhas joined
Ge0rG
Martin: that issue almost killed yax.im yesterday. Apparently internal storage is not fast at all.
jayteeukhas left
jayteeukhas joined
Ge0rG
Martin: there will be a solution to this soon, I'm sure.
ijhas left
Marandasuspects it's not about internal storage
Ge0rG
Maranda: it is. deleting one line from internal storage requires fully rewriting the huuuuuge serialized file
ackerman1scotthas left
ackerman1scotthas joined
kusonekohas left
kusonekohas joined
holgerhas left
kusonekohas left
kusonekohas joined
holgerhas joined
solhas left
solhas joined
solhas left
lorddavidiiihas left
Martin
Ge0rG: I was hoping that xmlstorage might help already. Maybe the structured data is easier to process.
Maranda
Ge0rG, that's not a problem here tbh, and me thinks it's more about that ".list" file there
ijhas joined
perflysthas left
solhas joined
Maranda
and that also you shouldn't keep to rewrite one huge serialized file to disk everytime just one line is popped
jayteeukhas left
jayteeukhas joined
insanityhas left
solhas left
solhas joined
insanityhas joined
lorddavidiiihas joined
MattJ
Ge0rG [18:49]:
> Martin: there will be a solution to this soon, I'm sure.
There is no real plan to fix it, it's been known that this does not perform well since before the release, which is why we tell people not to have large archives with internal storage
MattJ
It's possible that in a future release we will merge xmlarchive or a derivative
ackerman1scotthas left
MattJ
And maybe we can tweak e.g. removing more than one item at a time
MattJ
But rewriting how the archive store works on the stable branch is not on the cards, and would be a large project
solhas left
solhas joined
sakhmatdhas left
sakhmatdhas joined
kusonekohas left
kusonekohas joined
kusonekohas left
ackerman1scotthas joined
Ge0rG
MattJ: does this affect the stable branch? I've only encountered it after going to trunk
kusonekohas joined
sonnyhas left
sonnyhas joined
Ge0rG
and IIRC the 10k limit is the default, and it already makes prosody almost break down ;)
sonnyhas left
sonnyhas joined
MattJ
I thought it was in 0.11.x but not certain
colochonhas joined
kusonekohas left
kusonekohas joined
Ge0rG
MattJ: I'm sure I'd have run into it before then.
MattJ
And the default is not to persist MAM at all actually :)
kusonekohas left
kusonekohas joined
MattJ
(I don't believe in shipping software that logs peoples' messages by default)
kusonekohas left
kusonekohas joined
kusonekohas left
kusonekohas joined
Ge0rG
I don't believe in services that don't do everything they can to deliver messages