jonas’: looks like we have all the votes needed to issue two Last Calls :D
Jeybehas left
Jeybehas joined
goffihas left
goffihas joined
Alexhas joined
moparisthebest
I've just been informed that TCP connections are supposed to be short lived, opened, used to send or recieve data, then immediatly closed, and any application that does not use them in this manner is doing it wrong and shouldn't be expected to work
Daniel
by your ISP?
MattJ
Wow, thanks. Wish I'd known that sooner
jonas’
by whom?
moparisthebest
network team who installed some middleboxes that enforce this corporate-wide and is breaking a ton of stuff
moparisthebest
I too wish I had known sooner...
jonas’
my doctor just said that people with this type of opinion are a danger to public health due to high blood pressure and should be put in internet-restricted quarantine.
moparisthebest
I can't disagree jonas’
pep.
module:disable('s2s')
pep.
ah it's :unload
Zash
My sanity just said that those people need to be quarantined for life, preferably in Darvaza.
Zash
The middleboxes too.
flow
moparisthebest, did you ask them *why* those middleboxes enforce this?
moparisthebest
it's in the name of "network optimization" and "because that's the right way to use TCP"
moparisthebest
so far this has just been over email but I have a meeting in a few hours... -.-
Zash
Isn't this going to mess up email access too?
moparisthebest
our middleboxes are both riverbed and telari , I don't know which or both are doing this in particular, but might be something to watch out for
Zash
Or has everyone gone JMAP already?
MattJ
JMAP IDLE
moparisthebest
not sure what outlook uses actually
Zash
OWA maybe :(
Zash
Web eats everything :(
moparisthebest
I use the browser version and http does "use tcp correctly" so I guess I'm ok?
Zash
moparisthebest, you could tell them it breaks all your (paying customers) websocket applications
moparisthebest
we also have, nfs and cifs file servers too, guess we better turn them off
emushas joined
Jeybehas left
Jeybehas joined
Guus
kid just soiled itself, unsure if I can make board meeting
pep.
!
pep.
It's time indeed
pep.
Seve, MattJ ^
Sevesays hi
MattJ
o/
pep.
So we've got Guus and ralph excused
pep.
0. Agenda
pep.
Do we have anything for today?
MattJ
There were some items added to Trello
pep.
I'll leave ralph items for next week
pep.
1. Hire communications person
pep.
Quick update: I've poked peter, waiting for advice on contracting. Nothing else yet
pep.
2. AOB?
Seve
None for me, just thanks to iteam for bringing everything back up :)
pep.
Yep thanks a lot to iteam (MattJ) for handling the outage :)
pep.
There's an item for discussion in the board, "Iteam budget"
Yeah. So I opened a DigitalOcean account for the XSF, as a place to bring up a replacement server
MattJ
This is being billed from the XSF account since yesterday (thanks stpeter)
MattJ
Cost is pretty low, I don't remember which exact server I selected
MattJ
But I thought it would be good to have a clear budget for iteam
pep.
How much do you think you'd need?
MattJ
Because we may want to add additional servers as we modernize the infrastructure
Zash
And then jump on the k8s train‽
MattJ
I was thinking something like $1200/year
MattJ
As an upper limit, not a goal
Seve
Modernizing sounds very appealing to me
pep.
How many machines do you have with this weekend already? / cost
MattJ
Just the one server right now, checking the cost
pep.
Otherwise I'm happy to trust your decisions. $1200 seems very much fine
pep.
Previous hosting was being sponsored right?
pep.
Any news of them btw? Maybe different topic
MattJ
No news yet. Different but very related topic. Obviously their donation for all these years has basically kept our infrastructure costs at close to zero (we currently pay for tarsnap for backups)
MattJ
And moving stuff back there may make sense in the long run, but only if we can be sure this situation won't happen again
Zash
(AOB turned into an iteam report?)
pep.
Zash, kinda
MattJ
E.g. by providing us with remote access to cycle machines (which we used to have), and extra emergency contact details
pep.
But it's really another item :)
pep.
MattJ, for the budget, I agree, and I guess we'd all agree but we can talk about it again. Maybe bring that up on list? It seems we've got board members absent again in meetings
MattJ
Ok, current replacement server is $20.month
pep.
Okay
Seve
Setting a budged is a good idea anyway, so thanks for bringing this up MattJ
MattJ
If we're to expand, I can imagine adding a couple of smaller (cheaper) servers as well as this
MattJ
and maybe some third-party monitoring which would also have a cost
MattJ
But that's about it in my mind right now
pep.
Right. I don't think we'll get to $1200 right away, but it's also not really difficult if we wanted to :)
MattJ
I'm fine if we want to set the limit lower than that, and while we still have other servers donated, I think that would be fine
pep.
Ok, should we put this for next week to see if everybody agrees?
MattJ
wfm
MattJ
There was another item
pep.
k
MattJ
I thought it was on trello, but apparently not... it's to approve jonas’ joining iteam
pep.
That was 3. Iteam budget then
pep.
And 4. Another iteam
pep.
item, sorry.
MattJ
:)
pep.
Yep also sounds good to me
MattJ
jonas’ has kindly offered to help out with some of the modernization work
pep.
That team member list is also getting quite long :p
MattJ
Yeah, that's something that will be under review
pep.
Seve, thoughts?
MattJ
Figure out what servers we have -> who needs access to what -> who should/shouldn't be in the team
pep.
I really wished we were doing votes on-list as well (somewhat like council). Now we're just deferring until we get a fullhouse kinda
pep.
Anyway
Seve
Oh, I think I got disconnected
Seve
I'm happy for us moving in that direction, so nothing to complain about. Just maybe, I would prefer to have a "shorter" budget and increase it depending on our needs, just to make wiser decisions. But I completely trust Mat so all is good. Let's keep going on with this!
pep.
One doesn't have to use all the budget anyway :)
pep.
Ok, MattJ we're moving that also for next week then?
pep.
Or on-list if you can bring it there
MattJ
wfm
pep.
5. Any other AOB?
MattJ
I may do some planning/pricing and request a more specific budget, then post on list
pep.
Thanks
pep.
6. Close
pep.
Next: +1w
pep.
Thanks all
Marandahas left
Marandahas joined
MattJ
Thanks pep. :)
moparisthebest
> One doesn't have to use all the budget anyway :)
maybe you should talk to the government
Wojtekhas joined
adiaholic_has left
adiaholic_has joined
Jeybehas left
eevvoorhas left
Guus
I blame Frozen. She couldn't get out of her princess dress fast enough. Second time I had to clean the restroom today. The other kid was sick in it earlier.
Guus
MattJ: thanks for taking care of things. I'm not exactly sure on what happened in the first place. Would you mind sharing some details when you have some time?
MattJ
We have no details :(
MattJ
I don't even know what time the server went down
Guus
So, we 'just' have an unresponsive server and no way to get to it?
MattJ
Yes
MattJ
For what it's worth I found in my email archive that this also happened in March 2019
MattJ
And a power cycle brought it back to life
MattJ
But the guy who did that last year has not been responsive so far (and many DCs are locked down right now anyway)
lovetoxhas joined
Guus
To what extent do we have backups from which we can restore data, if need be?
MattJ
You're currently chatting on a server restored from backups
calvinhas joined
adiaholic_has left
MattJ
The backups are nightly, so we potentially lost up to 24h of chats/wiki edits, depending on exactly when it went down
Guus
Mail archives?
MattJ
Mail is on a different server
MattJ
(and that is also backed up)
Guus
In the same data center though?
MattJ
Yes
Guus
So the data center didn't explode. 😁
MattJ
Yeah, other stuff is working fine there (and the other servers are in the same rack)
Kev
Frankly, I think it's a wonder that Matt has managed to get 'everything' back up from a backup, given backups invariably turn out to not work when you need them. So despite being an inconvenience, I think this is a win!
MattJ
Yeah, this could have been a different story :) (one we've lived through before)
Guus
Hehe, glass half full kind of guy you are.
MattJ
So thanks Kev for leading that initiative back then
MattJ
Also, it's not necessarily the case that data on the down server is lost
MattJ
Just unreachable until it's powered back on
Kev
Although at least for wiki edits it effectively is.
MattJ
I don't plan to merge anything if it comes back, if that's what you mean
Kev
Because chances of us mananging to work out and reapply deltas between two DBs are limited.
Kev
Chat logs for a single day are probably easier to reinsert.
MattJ
But I wouldn't mind checking 'Recent changes' on the old server and comparing
Kev
Yeah, that wouldn't be daft.
MattJ
and a manual copy/paste in that case would suffice, likely
eevvoorhas joined
Guus
Thanks for taking care of everything. Let's find a way to prevent this from happening again in the future. If budget is needed for that, I think we should consider spending that.
debaclehas left
MattJ
It's a difficult situation - if we can continue to use donated hosting, I'm totally fine with that
Zash
Given that there were days between the crash and there store it should be relatively easy to merge the chat logs
Zash
Says me with my Prosody hat on, knowing how that's stored.
Zash
(hats!)
MattJ
We just need to ensure that things like this don't happen, and that we have an action plan when they do
Kev
I think I suggested something about this? Or did I dream that conversation?
Guus
Obviously, but I'd like to have a reasonable assurance that we can reach our servers when they're out. Last time, this was an issue too.
MattJ
I'm pretty sure if the XSF wanted to e.g. pay for a power strip that allows remote power cycling, that's already an improvement
MattJ
Kev, you did say similar in a private conversation with me
pep.
MattJ, I doubt you'd need this tbh
pep.
IPMI/ME should do, if available
pep.
and I guess that is, on a server
pep.
Now it does need to be plugged :p
MattJ
I /think/ the server in question doesn't have that, but I may be wrong
alexishas left
MattJ
Some of them do, but it's not exactly documented (except in ancient email threads)
alexishas joined
Kev
Typically we've bought hardware that does have ILO-ish things, but this box was different so I don't know.
Guus
There's probably a bunch of options that we can look at. I'm happy for iteam to first finish the disaster recovery, then come up with a plan for the future.
Guus
What's confusing to me is that in a DC, we're still limited to one contact person.
Guus
Isnt this a commercial DC, or are we serviced off the books or something?
MattJ
Our relationship with them pre-dates my involvement with the XSF, so I don't know the details
Kev
This is all done out of the goodness of Jerry's heart, and has been since the year Dot. We're not a client or anything.
Guus
Although I'm very happy for Jerry's efforts, some kind of more robust service might be preferable, given that our hardware seems to become less reliable.
Guus
I'm thinking we are running on aging hardware?
Kev
That's also because we live off donated and/or ancient hardware etc., mind.
Guus
Sure, it is totally understandable
Guus
We have a pretty good deal on all this
Kev
I remember exactly one issue with the hosting, as opposed to the machines -and far more instances of the opposite, where Jerry's just found 'random' hard drives/other hardware to patch up our servers to get them straight back up.
Kev
(And that issue with the hosting was because we were getting DoSd so Jerry had to DoS protection stuff)
MattJ
Another part of me thinks that we don't /need/ dedicated hardware for what we do these days. Maybe in 2004, but virtual servers are cheap and easy these days.
Guus
But maybe it's time to improve reliability by spending a bit more money
Kev
So I'm very happy to recommend various changes, but I don't think our hosting environment is a significant part of our issues.
archas left
archas joined
Kev
Yes, we could probably (other than xmpp.net) cope with a £5/mo servers4u VPS or something.
Guus
If an important server is unreachable for several days, I think it is somewhat of a problem.
pep.
MattJ, depends what we want to do with them
pdurbinhas joined
MattJ
It's been a week now
MattJ
(well, it went offline some time between Thu night and Fri morning)
archas left
archas joined
lovetoxhas left
pdurbinhas left
archas left
archas joined
archas left
archas joined
krauqhas left
krauqhas joined
sonnyhas left
eevvoorhas left
werdanhas joined
adiaholic_has joined
lovetoxhas joined
j.rhas left
j.rhas joined
edhelashas left
edhelashas joined
Jeybehas joined
adiaholic_has left
adiaholic_has joined
mukt2has joined
mukt2has left
goffihas left
Steve Killehas left
goffihas joined
Steve Killehas joined
etahas left
etahas joined
archas left
LNJhas left
archas joined
archas left
archas joined
Yagizahas left
LNJhas joined
eevvoorhas joined
sonnyhas joined
João Duartehas joined
waqashas joined
alexishas left
Jeybehas left
Jeybehas joined
Steve Killehas left
mukt2has joined
Jeybehas left
Robertohas left
Steve Killehas joined
Tobiashas left
Tobiashas joined
archas left
archas joined
Alexhas left
Alexhas joined
archas left
archas joined
João Duarte
I just realized now, after 4 days, that my clients werent updating any messages
Half-Shothas left
Half-Shothas joined
Alexhas left
Alexhas joined
João Duarte
Was coming here to give Manav some feedback, but now if i understood correctly, it was some server issue that is causing trouble to the chat rooms?
mukt2has left
flow
João Duarte, yep
João Duarte
Ah okay. Anyway to get things back updating in the clients? Or just wait?
Zash
Depends on the client, but leaving and rejoining should do the trick
Alexhas left
Alexhas joined
João Duarte
Im using pidgin on desktop and xabber on mobile
João Duarte
Already tried rejoining to no avail
João Duarte
But maybe i can troubleshoot my way through, just wanted to check it was within my reach and not something purely on the server
Zash
The server that served this chat went down and we don't know why. This chat is now running on a new server.
João Duarte
Same thing for prosody's chat?
Zash
No
Alexhas left
Alexhas joined
João Duarte
I guess so, because the last messages I have were sent in the 22th
João Duarte
hm
Zash
Then it was probably not the XSF server issue.
Link Mauve
jonas’, you once told me not to include real client names in XEPs right? I’m fixing MIX’s examples which were violating XEP-0030 MUSTs.
Link Mauve
Ah no, in the end you accepted it in XEP-0402.
Alexhas left
Alexhas joined
adiaholic_has left
adiaholic_has joined
João Duartehas left
edhelashas left
edhelashas joined
Jeybehas joined
Nekithas left
Jeybehas left
Jeybehas joined
remkohas joined
Link Mauve
“A nick MAY be specified as a <nick/> child elements of the <join/> element.”, what happens if the user’s server didn’t specify a nick and the MIX channel requires one?
Ge0rG
Link Mauve: yes.
Wojtekhas left
Nekithas joined
Link Mauve
Ah, “If nick is missing on a channel where nick is mandatory, the join MUST be rejected.”, nvm.
Wojtekhas joined
Robertohas joined
Jeybehas left
adiaholic_has left
mukt2has joined
pdurbinhas joined
Link Mauve
Again in MIX, why is it <update-subscription/> but <setnick/>?
eevvoorhas left
eevvoorhas joined
mukt2has left
werdanhas left
Link Mauve
In section 7.3.4, this is the first time there is no <{mix}destroy/> in the iq result, is this wanted?