XSF Discussion - 2020-03-26

jonas’: looks like we have all the votes needed to issue two Last Calls :D

I've just been informed that TCP connections are supposed to be short lived, opened, used to send or recieve data, then immediatly closed, and any application that does not use them in this manner is doing it wrong and shouldn't be expected to work

by your ISP?

Wow, thanks. Wish I'd known that sooner

by whom?

network team who installed some middleboxes that enforce this corporate-wide and is breaking a ton of stuff

I too wish I had known sooner...

my doctor just said that people with this type of opinion are a danger to public health due to high blood pressure and should be put in internet-restricted quarantine.

I can't disagree jonas’

module:disable('s2s')

ah it's :unload

My sanity just said that those people need to be quarantined for life, preferably in Darvaza.

The middleboxes too.

moparisthebest, did you ask them *why* those middleboxes enforce this?

it's in the name of "network optimization" and "because that's the right way to use TCP"

so far this has just been over email but I have a meeting in a few hours... -.-

Isn't this going to mess up email access too?

our middleboxes are both riverbed and telari , I don't know which or both are doing this in particular, but might be something to watch out for

Or has everyone gone JMAP already?

JMAP IDLE

not sure what outlook uses actually

OWA maybe :(

Web eats everything :(

I use the browser version and http does "use tcp correctly" so I guess I'm ok?

moparisthebest, you could tell them it breaks all your (paying customers) websocket applications

we also have, nfs and cifs file servers too, guess we better turn them off

kid just soiled itself, unsure if I can make board meeting

!

It's time indeed

Seve, MattJ ^

o/

So we've got Guus and ralph excused

0. Agenda

Do we have anything for today?

There were some items added to Trello

I'll leave ralph items for next week

1. Hire communications person

Quick update: I've poked peter, waiting for advice on contracting. Nothing else yet

2. AOB?

None for me, just thanks to iteam for bringing everything back up :)

Yep thanks a lot to iteam (MattJ) for handling the outage :)

There's an item for discussion in the board, "Iteam budget"

MattJ, if you want to talk about it?

MattJ, might want to expand that ✏

Yeah. So I opened a DigitalOcean account for the XSF, as a place to bring up a replacement server

This is being billed from the XSF account since yesterday (thanks stpeter)

Cost is pretty low, I don't remember which exact server I selected

But I thought it would be good to have a clear budget for iteam

How much do you think you'd need?

Because we may want to add additional servers as we modernize the infrastructure

And then jump on the k8s train‽

I was thinking something like $1200/year

As an upper limit, not a goal

Modernizing sounds very appealing to me

How many machines do you have with this weekend already? / cost

Just the one server right now, checking the cost

Otherwise I'm happy to trust your decisions. $1200 seems very much fine

Previous hosting was being sponsored right?

Any news of them btw? Maybe different topic

No news yet. Different but very related topic. Obviously their donation for all these years has basically kept our infrastructure costs at close to zero (we currently pay for tarsnap for backups)

And moving stuff back there may make sense in the long run, but only if we can be sure this situation won't happen again

(AOB turned into an iteam report?)

Zash, kinda

E.g. by providing us with remote access to cycle machines (which we used to have), and extra emergency contact details

But it's really another item :)

MattJ, for the budget, I agree, and I guess we'd all agree but we can talk about it again. Maybe bring that up on list? It seems we've got board members absent again in meetings

Ok, current replacement server is $20.month

Okay

Setting a budged is a good idea anyway, so thanks for bringing this up MattJ

If we're to expand, I can imagine adding a couple of smaller (cheaper) servers as well as this

and maybe some third-party monitoring which would also have a cost

But that's about it in my mind right now

Right. I don't think we'll get to $1200 right away, but it's also not really difficult if we wanted to :)

I'm fine if we want to set the limit lower than that, and while we still have other servers donated, I think that would be fine

Ok, should we put this for next week to see if everybody agrees?

wfm

There was another item

k

I thought it was on trello, but apparently not... it's to approve jonas’ joining iteam

That was 3. Iteam budget then

And 4. Another iteam

item, sorry.

:)

Yep also sounds good to me

jonas’ has kindly offered to help out with some of the modernization work

That team member list is also getting quite long :p

Yeah, that's something that will be under review

Seve, thoughts?

Figure out what servers we have -> who needs access to what -> who should/shouldn't be in the team

I really wished we were doing votes on-list as well (somewhat like council). Now we're just deferring until we get a fullhouse kinda

Anyway

Oh, I think I got disconnected

I'm happy for us moving in that direction, so nothing to complain about. Just maybe, I would prefer to have a "shorter" budget and increase it depending on our needs, just to make wiser decisions. But I completely trust Mat so all is good. Let's keep going on with this!

One doesn't have to use all the budget anyway :)

Ok, MattJ we're moving that also for next week then?

Or on-list if you can bring it there

wfm

5. Any other AOB?

I may do some planning/pricing and request a more specific budget, then post on list

Thanks

6. Close

Next: +1w

Thanks all

Thanks pep. :)

> One doesn't have to use all the budget anyway :) maybe you should talk to the government

I blame Frozen. She couldn't get out of her princess dress fast enough. Second time I had to clean the restroom today. The other kid was sick in it earlier.

MattJ: thanks for taking care of things. I'm not exactly sure on what happened in the first place. Would you mind sharing some details when you have some time?

We have no details :(

I don't even know what time the server went down

So, we 'just' have an unresponsive server and no way to get to it?

Yes

For what it's worth I found in my email archive that this also happened in March 2019

And a power cycle brought it back to life

But the guy who did that last year has not been responsive so far (and many DCs are locked down right now anyway)

To what extent do we have backups from which we can restore data, if need be?

You're currently chatting on a server restored from backups

The backups are nightly, so we potentially lost up to 24h of chats/wiki edits, depending on exactly when it went down

Mail archives?

Mail is on a different server

(and that is also backed up)

In the same data center though?

Yes

So the data center didn't explode. 😁

Yeah, other stuff is working fine there (and the other servers are in the same rack)

Frankly, I think it's a wonder that Matt has managed to get 'everything' back up from a backup, given backups invariably turn out to not work when you need them. So despite being an inconvenience, I think this is a win!

Yeah, this could have been a different story :) (one we've lived through before)

Hehe, glass half full kind of guy you are.

So thanks Kev for leading that initiative back then

Also, it's not necessarily the case that data on the down server is lost

Just unreachable until it's powered back on

Although at least for wiki edits it effectively is.

I don't plan to merge anything if it comes back, if that's what you mean

Because chances of us mananging to work out and reapply deltas between two DBs are limited.

Chat logs for a single day are probably easier to reinsert.

But I wouldn't mind checking 'Recent changes' on the old server and comparing

Yeah, that wouldn't be daft.

and a manual copy/paste in that case would suffice, likely

Thanks for taking care of everything. Let's find a way to prevent this from happening again in the future. If budget is needed for that, I think we should consider spending that.

It's a difficult situation - if we can continue to use donated hosting, I'm totally fine with that

Given that there were days between the crash and there store it should be relatively easy to merge the chat logs

Says me with my Prosody hat on, knowing how that's stored.

(hats!)

We just need to ensure that things like this don't happen, and that we have an action plan when they do

I think I suggested something about this? Or did I dream that conversation?

Obviously, but I'd like to have a reasonable assurance that we can reach our servers when they're out. Last time, this was an issue too.

I'm pretty sure if the XSF wanted to e.g. pay for a power strip that allows remote power cycling, that's already an improvement

Kev, you did say similar in a private conversation with me

MattJ, I doubt you'd need this tbh

IPMI/ME should do, if available

and I guess that is, on a server

Now it does need to be plugged :p

I /think/ the server in question doesn't have that, but I may be wrong

Some of them do, but it's not exactly documented (except in ancient email threads)

Typically we've bought hardware that does have ILO-ish things, but this box was different so I don't know.

There's probably a bunch of options that we can look at. I'm happy for iteam to first finish the disaster recovery, then come up with a plan for the future.

What's confusing to me is that in a DC, we're still limited to one contact person.

Isnt this a commercial DC, or are we serviced off the books or something?

Our relationship with them pre-dates my involvement with the XSF, so I don't know the details

This is all done out of the goodness of Jerry's heart, and has been since the year Dot. We're not a client or anything.

Although I'm very happy for Jerry's efforts, some kind of more robust service might be preferable, given that our hardware seems to become less reliable.

I'm thinking we are running on aging hardware?

That's also because we live off donated and/or ancient hardware etc., mind.

Sure, it is totally understandable

We have a pretty good deal on all this

I remember exactly one issue with the hosting, as opposed to the machines -and far more instances of the opposite, where Jerry's just found 'random' hard drives/other hardware to patch up our servers to get them straight back up.

(And that issue with the hosting was because we were getting DoSd so Jerry had to DoS protection stuff)

Another part of me thinks that we don't /need/ dedicated hardware for what we do these days. Maybe in 2004, but virtual servers are cheap and easy these days.

But maybe it's time to improve reliability by spending a bit more money

So I'm very happy to recommend various changes, but I don't think our hosting environment is a significant part of our issues.

Yes, we could probably (other than xmpp.net) cope with a £5/mo servers4u VPS or something.

If an important server is unreachable for several days, I think it is somewhat of a problem.

MattJ, depends what we want to do with them

It's been a week now

(well, it went offline some time between Thu night and Fri morning)

I just realized now, after 4 days, that my clients werent updating any messages

Was coming here to give Manav some feedback, but now if i understood correctly, it was some server issue that is causing trouble to the chat rooms?

João Duarte, yep

Ah okay. Anyway to get things back updating in the clients? Or just wait?

Depends on the client, but leaving and rejoining should do the trick

Im using pidgin on desktop and xabber on mobile

Already tried rejoining to no avail

But maybe i can troubleshoot my way through, just wanted to check it was within my reach and not something purely on the server

The server that served this chat went down and we don't know why. This chat is now running on a new server.

Same thing for prosody's chat?

No

I guess so, because the last messages I have were sent in the 22th

hm

Then it was probably not the XSF server issue.

jonas’, you once told me not to include real client names in XEPs right? I’m fixing MIX’s examples which were violating XEP-0030 MUSTs.

Ah no, in the end you accepted it in XEP-0402.

“A nick MAY be specified as a <nick/> child elements of the <join/> element.”, what happens if the user’s server didn’t specify a nick and the MIX channel requires one?

Link Mauve: yes.

Ah, “If nick is missing on a channel where nick is mandatory, the join MUST be rejected.”, nvm.

Again in MIX, why is it <update-subscription/> but <setnick/>?

In section 7.3.4, this is the first time there is no <{mix}destroy/> in the iq result, is this wanted?

MIX is missing error cases.