-
Ge0rG
jonas’: looks like we have all the votes needed to issue two Last Calls :D
-
moparisthebest
I've just been informed that TCP connections are supposed to be short lived, opened, used to send or recieve data, then immediatly closed, and any application that does not use them in this manner is doing it wrong and shouldn't be expected to work
-
Daniel
by your ISP?
-
MattJ
Wow, thanks. Wish I'd known that sooner
-
jonas’
by whom?
-
moparisthebest
network team who installed some middleboxes that enforce this corporate-wide and is breaking a ton of stuff
-
moparisthebest
I too wish I had known sooner...
-
jonas’
my doctor just said that people with this type of opinion are a danger to public health due to high blood pressure and should be put in internet-restricted quarantine.
-
moparisthebest
I can't disagree jonas’
-
pep.
module:disable('s2s')
-
pep.
ah it's :unload
-
Zash
My sanity just said that those people need to be quarantined for life, preferably in Darvaza.
-
Zash
The middleboxes too.
-
flow
moparisthebest, did you ask them *why* those middleboxes enforce this?
-
moparisthebest
it's in the name of "network optimization" and "because that's the right way to use TCP"
-
moparisthebest
so far this has just been over email but I have a meeting in a few hours... -.-
-
Zash
Isn't this going to mess up email access too?
-
moparisthebest
our middleboxes are both riverbed and telari , I don't know which or both are doing this in particular, but might be something to watch out for
-
Zash
Or has everyone gone JMAP already?
-
MattJ
JMAP IDLE
-
moparisthebest
not sure what outlook uses actually
-
Zash
OWA maybe :(
-
Zash
Web eats everything :(
-
moparisthebest
I use the browser version and http does "use tcp correctly" so I guess I'm ok?
-
Zash
moparisthebest, you could tell them it breaks all your (paying customers) websocket applications
-
moparisthebest
we also have, nfs and cifs file servers too, guess we better turn them off
-
Guus
kid just soiled itself, unsure if I can make board meeting
-
pep.
!
-
pep.
It's time indeed
-
pep.
Seve, MattJ ^
- Seve says hi
-
MattJ
o/
-
pep.
So we've got Guus and ralph excused
-
pep.
0. Agenda
-
pep.
Do we have anything for today?
-
MattJ
There were some items added to Trello
-
pep.
I'll leave ralph items for next week
-
pep.
1. Hire communications person
-
pep.
Quick update: I've poked peter, waiting for advice on contracting. Nothing else yet
-
pep.
2. AOB?
-
Seve
None for me, just thanks to iteam for bringing everything back up :)
-
pep.
Yep thanks a lot to iteam (MattJ) for handling the outage :)
-
pep.
There's an item for discussion in the board, "Iteam budget"
-
pep.
MattJ, if you want to talk about it?
-
Seve
MattJ, might want to expan that✎ -
Seve
MattJ, might want to expand that ✏
-
MattJ
Yeah. So I opened a DigitalOcean account for the XSF, as a place to bring up a replacement server
-
MattJ
This is being billed from the XSF account since yesterday (thanks stpeter)
-
MattJ
Cost is pretty low, I don't remember which exact server I selected
-
MattJ
But I thought it would be good to have a clear budget for iteam
-
pep.
How much do you think you'd need?
-
MattJ
Because we may want to add additional servers as we modernize the infrastructure
-
Zash
And then jump on the k8s train‽
-
MattJ
I was thinking something like $1200/year
-
MattJ
As an upper limit, not a goal
-
Seve
Modernizing sounds very appealing to me
-
pep.
How many machines do you have with this weekend already? / cost
-
MattJ
Just the one server right now, checking the cost
-
pep.
Otherwise I'm happy to trust your decisions. $1200 seems very much fine
-
pep.
Previous hosting was being sponsored right?
-
pep.
Any news of them btw? Maybe different topic
-
MattJ
No news yet. Different but very related topic. Obviously their donation for all these years has basically kept our infrastructure costs at close to zero (we currently pay for tarsnap for backups)
-
MattJ
And moving stuff back there may make sense in the long run, but only if we can be sure this situation won't happen again
-
Zash
(AOB turned into an iteam report?)
-
pep.
Zash, kinda
-
MattJ
E.g. by providing us with remote access to cycle machines (which we used to have), and extra emergency contact details
-
pep.
But it's really another item :)
-
pep.
MattJ, for the budget, I agree, and I guess we'd all agree but we can talk about it again. Maybe bring that up on list? It seems we've got board members absent again in meetings
-
MattJ
Ok, current replacement server is $20.month
-
pep.
Okay
-
Seve
Setting a budged is a good idea anyway, so thanks for bringing this up MattJ
-
MattJ
If we're to expand, I can imagine adding a couple of smaller (cheaper) servers as well as this
-
MattJ
and maybe some third-party monitoring which would also have a cost
-
MattJ
But that's about it in my mind right now
-
pep.
Right. I don't think we'll get to $1200 right away, but it's also not really difficult if we wanted to :)
-
MattJ
I'm fine if we want to set the limit lower than that, and while we still have other servers donated, I think that would be fine
-
pep.
Ok, should we put this for next week to see if everybody agrees?
-
MattJ
wfm
-
MattJ
There was another item
-
pep.
k
-
MattJ
I thought it was on trello, but apparently not... it's to approve jonas’ joining iteam
-
pep.
That was 3. Iteam budget then
-
pep.
And 4. Another iteam
-
pep.
item, sorry.
-
MattJ
:)
-
pep.
Yep also sounds good to me
-
MattJ
jonas’ has kindly offered to help out with some of the modernization work
-
pep.
That team member list is also getting quite long :p
-
MattJ
Yeah, that's something that will be under review
-
pep.
Seve, thoughts?
-
MattJ
Figure out what servers we have -> who needs access to what -> who should/shouldn't be in the team
-
pep.
I really wished we were doing votes on-list as well (somewhat like council). Now we're just deferring until we get a fullhouse kinda
-
pep.
Anyway
-
Seve
Oh, I think I got disconnected
-
Seve
I'm happy for us moving in that direction, so nothing to complain about. Just maybe, I would prefer to have a "shorter" budget and increase it depending on our needs, just to make wiser decisions. But I completely trust Mat so all is good. Let's keep going on with this!
-
pep.
One doesn't have to use all the budget anyway :)
-
pep.
Ok, MattJ we're moving that also for next week then?
-
pep.
Or on-list if you can bring it there
-
MattJ
wfm
-
pep.
5. Any other AOB?
-
MattJ
I may do some planning/pricing and request a more specific budget, then post on list
-
pep.
Thanks
-
pep.
6. Close
-
pep.
Next: +1w
-
pep.
Thanks all
-
MattJ
Thanks pep. :)
-
moparisthebest
> One doesn't have to use all the budget anyway :) maybe you should talk to the government
-
Guus
I blame Frozen. She couldn't get out of her princess dress fast enough. Second time I had to clean the restroom today. The other kid was sick in it earlier.
-
Guus
MattJ: thanks for taking care of things. I'm not exactly sure on what happened in the first place. Would you mind sharing some details when you have some time?
-
MattJ
We have no details :(
-
MattJ
I don't even know what time the server went down
-
Guus
So, we 'just' have an unresponsive server and no way to get to it?
-
MattJ
Yes
-
MattJ
For what it's worth I found in my email archive that this also happened in March 2019
-
MattJ
And a power cycle brought it back to life
-
MattJ
But the guy who did that last year has not been responsive so far (and many DCs are locked down right now anyway)
-
Guus
To what extent do we have backups from which we can restore data, if need be?
-
MattJ
You're currently chatting on a server restored from backups
-
MattJ
The backups are nightly, so we potentially lost up to 24h of chats/wiki edits, depending on exactly when it went down
-
Guus
Mail archives?
-
MattJ
Mail is on a different server
-
MattJ
(and that is also backed up)
-
Guus
In the same data center though?
-
MattJ
Yes
-
Guus
So the data center didn't explode. 😁
-
MattJ
Yeah, other stuff is working fine there (and the other servers are in the same rack)
-
Kev
Frankly, I think it's a wonder that Matt has managed to get 'everything' back up from a backup, given backups invariably turn out to not work when you need them. So despite being an inconvenience, I think this is a win!
-
MattJ
Yeah, this could have been a different story :) (one we've lived through before)
-
Guus
Hehe, glass half full kind of guy you are.
-
MattJ
So thanks Kev for leading that initiative back then
-
MattJ
Also, it's not necessarily the case that data on the down server is lost
-
MattJ
Just unreachable until it's powered back on
-
Kev
Although at least for wiki edits it effectively is.
-
MattJ
I don't plan to merge anything if it comes back, if that's what you mean
-
Kev
Because chances of us mananging to work out and reapply deltas between two DBs are limited.
-
Kev
Chat logs for a single day are probably easier to reinsert.
-
MattJ
But I wouldn't mind checking 'Recent changes' on the old server and comparing
-
Kev
Yeah, that wouldn't be daft.
-
MattJ
and a manual copy/paste in that case would suffice, likely
-
Guus
Thanks for taking care of everything. Let's find a way to prevent this from happening again in the future. If budget is needed for that, I think we should consider spending that.
-
MattJ
It's a difficult situation - if we can continue to use donated hosting, I'm totally fine with that
-
Zash
Given that there were days between the crash and there store it should be relatively easy to merge the chat logs
-
Zash
Says me with my Prosody hat on, knowing how that's stored.
-
Zash
(hats!)
-
MattJ
We just need to ensure that things like this don't happen, and that we have an action plan when they do
-
Kev
I think I suggested something about this? Or did I dream that conversation?
-
Guus
Obviously, but I'd like to have a reasonable assurance that we can reach our servers when they're out. Last time, this was an issue too.
-
MattJ
I'm pretty sure if the XSF wanted to e.g. pay for a power strip that allows remote power cycling, that's already an improvement
-
MattJ
Kev, you did say similar in a private conversation with me
-
pep.
MattJ, I doubt you'd need this tbh
-
pep.
IPMI/ME should do, if available
-
pep.
and I guess that is, on a server
-
pep.
Now it does need to be plugged :p
-
MattJ
I /think/ the server in question doesn't have that, but I may be wrong
-
MattJ
Some of them do, but it's not exactly documented (except in ancient email threads)
-
Kev
Typically we've bought hardware that does have ILO-ish things, but this box was different so I don't know.
-
Guus
There's probably a bunch of options that we can look at. I'm happy for iteam to first finish the disaster recovery, then come up with a plan for the future.
-
Guus
What's confusing to me is that in a DC, we're still limited to one contact person.
-
Guus
Isnt this a commercial DC, or are we serviced off the books or something?
-
MattJ
Our relationship with them pre-dates my involvement with the XSF, so I don't know the details
-
Kev
This is all done out of the goodness of Jerry's heart, and has been since the year Dot. We're not a client or anything.
-
Guus
Although I'm very happy for Jerry's efforts, some kind of more robust service might be preferable, given that our hardware seems to become less reliable.
-
Guus
I'm thinking we are running on aging hardware?
-
Kev
That's also because we live off donated and/or ancient hardware etc., mind.
-
Guus
Sure, it is totally understandable
-
Guus
We have a pretty good deal on all this
-
Kev
I remember exactly one issue with the hosting, as opposed to the machines -and far more instances of the opposite, where Jerry's just found 'random' hard drives/other hardware to patch up our servers to get them straight back up.
-
Kev
(And that issue with the hosting was because we were getting DoSd so Jerry had to DoS protection stuff)
-
MattJ
Another part of me thinks that we don't /need/ dedicated hardware for what we do these days. Maybe in 2004, but virtual servers are cheap and easy these days.
-
Guus
But maybe it's time to improve reliability by spending a bit more money
-
Kev
So I'm very happy to recommend various changes, but I don't think our hosting environment is a significant part of our issues.
-
Kev
Yes, we could probably (other than xmpp.net) cope with a £5/mo servers4u VPS or something.
-
Guus
If an important server is unreachable for several days, I think it is somewhat of a problem.
-
pep.
MattJ, depends what we want to do with them
-
MattJ
It's been a week now
-
MattJ
(well, it went offline some time between Thu night and Fri morning)
-
João Duarte
I just realized now, after 4 days, that my clients werent updating any messages
-
João Duarte
Was coming here to give Manav some feedback, but now if i understood correctly, it was some server issue that is causing trouble to the chat rooms?
-
flow
João Duarte, yep
-
João Duarte
Ah okay. Anyway to get things back updating in the clients? Or just wait?
-
Zash
Depends on the client, but leaving and rejoining should do the trick
-
João Duarte
Im using pidgin on desktop and xabber on mobile
-
João Duarte
Already tried rejoining to no avail
-
João Duarte
But maybe i can troubleshoot my way through, just wanted to check it was within my reach and not something purely on the server
-
Zash
The server that served this chat went down and we don't know why. This chat is now running on a new server.
-
João Duarte
Same thing for prosody's chat?
-
Zash
No
-
João Duarte
I guess so, because the last messages I have were sent in the 22th
-
João Duarte
hm
-
Zash
Then it was probably not the XSF server issue.
-
Link Mauve
jonas’, you once told me not to include real client names in XEPs right? I’m fixing MIX’s examples which were violating XEP-0030 MUSTs.
-
Link Mauve
Ah no, in the end you accepted it in XEP-0402.
-
Link Mauve
“A nick MAY be specified as a <nick/> child elements of the <join/> element.”, what happens if the user’s server didn’t specify a nick and the MIX channel requires one?
-
Ge0rG
Link Mauve: yes.
-
Link Mauve
Ah, “If nick is missing on a channel where nick is mandatory, the join MUST be rejected.”, nvm.
-
Link Mauve
Again in MIX, why is it <update-subscription/> but <setnick/>?
-
Link Mauve
In section 7.3.4, this is the first time there is no <{mix}destroy/> in the iq result, is this wanted?
-
Link Mauve
MIX is missing error cases.