XMPP Service Operators - 2021-03-30

jonas’: I got o.j.n reports at 09:07 and 09:12 but I don't see anything in my logs. If you are free and it doesn't make too much effort might you have a look what caused them. I am really clueless.

https://files.mdosch.de:5281/upload/kxJJ3xeOt6uqDKsS/2021-03-30-103332_scrot.png

I got prober infrastructure related alerts around the same time

and BCCs of yax.im alerts

yax.im had downtime issues today

I rebooted the server short after 9 CEST

Martin, yeah, both prober nodes had issues, looks like an outage of the monitoring itself.

Ge0rG, I sure hope that was not related.

jonas’: should I check my logs for whether ojn killed the server? ;)

I think it was rather related to /proc being dysfunction.

Ge0rG, please validate when exactly you restarted

10:39:05 up 1:20, 1 user, load average: 0.18, 0.23, 0.19

that would make it 09:19 CEST?

Yeah.

that’s 5 minutes after the first ojn prober node alert came ✏

this is really strange

good thing I have a kibana to look at this later on

jonas’: Right. I reacted to the prober alert by switching screen terminals in SSH and everything was laggy and felt b0rked. Well possible that there was some global carrier outage

But as the machine was in a weird state anyway I decided to reboot.

that would be an interesting outage because both vantage points reported >50% error rate

(hetzner-AS and conova-AS)

and it affected at least you and martin

I’ll have to take a careful look at everything later

jonas’: I don't have any anomalies on the icmp graphs from hosteurope

jonas’, Ge0rG: Thanks for the clarification.

Ge0rG, Martin, one of you folks running with debug logs enabled, can you check if the prober ojn still sends garbage after </stream:stream> or whether that also magically disappeared with the update I just did?

No debug logs here

So the prober sometimes sent garbage? Was this the reason for the this mornings alarms? Then I wonder why it didn't happen earlier or more often. 🤔