Ge0rGjonas’: looks like we have all the votes needed to issue two Last Calls :D
Jeybehas left
Jeybehas joined
goffihas left
goffihas joined
Alexhas joined
moparisthebestI've just been informed that TCP connections are supposed to be short lived, opened, used to send or recieve data, then immediatly closed, and any application that does not use them in this manner is doing it wrong and shouldn't be expected to work
Danielby your ISP?
MattJWow, thanks. Wish I'd known that sooner
jonas’by whom?
moparisthebestnetwork team who installed some middleboxes that enforce this corporate-wide and is breaking a ton of stuff
moparisthebestI too wish I had known sooner...
jonas’my doctor just said that people with this type of opinion are a danger to public health due to high blood pressure and should be put in internet-restricted quarantine.
moparisthebestI can't disagree jonas’
pep.module:disable('s2s')
pep.ah it's :unload
ZashMy sanity just said that those people need to be quarantined for life, preferably in Darvaza.
ZashThe middleboxes too.
flowmoparisthebest, did you ask them *why* those middleboxes enforce this?
moparisthebestit's in the name of "network optimization" and "because that's the right way to use TCP"
moparisthebestso far this has just been over email but I have a meeting in a few hours... -.-
ZashIsn't this going to mess up email access too?
moparisthebestour middleboxes are both riverbed and telari , I don't know which or both are doing this in particular, but might be something to watch out for
ZashOr has everyone gone JMAP already?
MattJJMAP IDLE
moparisthebestnot sure what outlook uses actually
ZashOWA maybe :(
ZashWeb eats everything :(
moparisthebestI use the browser version and http does "use tcp correctly" so I guess I'm ok?
Zashmoparisthebest, you could tell them it breaks all your (paying customers) websocket applications
moparisthebestwe also have, nfs and cifs file servers too, guess we better turn them off
emushas joined
Jeybehas left
Jeybehas joined
Guuskid just soiled itself, unsure if I can make board meeting
pep.!
pep.It's time indeed
pep.Seve, MattJ ^
Sevesays hi
MattJo/
pep.So we've got Guus and ralph excused
pep.0. Agenda
pep.Do we have anything for today?
MattJThere were some items added to Trello
pep.I'll leave ralph items for next week
pep.1. Hire communications person
pep.Quick update: I've poked peter, waiting for advice on contracting. Nothing else yet
pep.2. AOB?
SeveNone for me, just thanks to iteam for bringing everything back up :)
pep.Yep thanks a lot to iteam (MattJ) for handling the outage :)
pep.There's an item for discussion in the board, "Iteam budget"
pep.MattJ, if you want to talk about it?
SeveMattJ, might want to expan that
SeveMattJ, might want to expand that
MattJYeah. So I opened a DigitalOcean account for the XSF, as a place to bring up a replacement server
MattJThis is being billed from the XSF account since yesterday (thanks stpeter)
MattJCost is pretty low, I don't remember which exact server I selected
MattJBut I thought it would be good to have a clear budget for iteam
pep.How much do you think you'd need?
MattJBecause we may want to add additional servers as we modernize the infrastructure
ZashAnd then jump on the k8s train‽
MattJI was thinking something like $1200/year
MattJAs an upper limit, not a goal
SeveModernizing sounds very appealing to me
pep.How many machines do you have with this weekend already? / cost
MattJJust the one server right now, checking the cost
pep.Otherwise I'm happy to trust your decisions. $1200 seems very much fine
pep.Previous hosting was being sponsored right?
pep.Any news of them btw? Maybe different topic
MattJNo news yet. Different but very related topic. Obviously their donation for all these years has basically kept our infrastructure costs at close to zero (we currently pay for tarsnap for backups)
MattJAnd moving stuff back there may make sense in the long run, but only if we can be sure this situation won't happen again
Zash(AOB turned into an iteam report?)
pep.Zash, kinda
MattJE.g. by providing us with remote access to cycle machines (which we used to have), and extra emergency contact details
pep.But it's really another item :)
pep.MattJ, for the budget, I agree, and I guess we'd all agree but we can talk about it again. Maybe bring that up on list? It seems we've got board members absent again in meetings
MattJOk, current replacement server is $20.month
pep.Okay
SeveSetting a budged is a good idea anyway, so thanks for bringing this up MattJ
MattJIf we're to expand, I can imagine adding a couple of smaller (cheaper) servers as well as this
MattJand maybe some third-party monitoring which would also have a cost
MattJBut that's about it in my mind right now
pep.Right. I don't think we'll get to $1200 right away, but it's also not really difficult if we wanted to :)
MattJI'm fine if we want to set the limit lower than that, and while we still have other servers donated, I think that would be fine
pep.Ok, should we put this for next week to see if everybody agrees?
MattJwfm
MattJThere was another item
pep.k
MattJI thought it was on trello, but apparently not... it's to approve jonas’ joining iteam
pep.That was 3. Iteam budget then
pep.And 4. Another iteam
pep.item, sorry.
MattJ:)
pep.Yep also sounds good to me
MattJjonas’ has kindly offered to help out with some of the modernization work
pep.That team member list is also getting quite long :p
MattJYeah, that's something that will be under review
pep.Seve, thoughts?
MattJFigure out what servers we have -> who needs access to what -> who should/shouldn't be in the team
pep.I really wished we were doing votes on-list as well (somewhat like council). Now we're just deferring until we get a fullhouse kinda
pep.Anyway
SeveOh, I think I got disconnected
SeveI'm happy for us moving in that direction, so nothing to complain about. Just maybe, I would prefer to have a "shorter" budget and increase it depending on our needs, just to make wiser decisions. But I completely trust Mat so all is good. Let's keep going on with this!
pep.One doesn't have to use all the budget anyway :)
pep.Ok, MattJ we're moving that also for next week then?
pep.Or on-list if you can bring it there
MattJwfm
pep.5. Any other AOB?
MattJI may do some planning/pricing and request a more specific budget, then post on list
pep.Thanks
pep.6. Close
pep.Next: +1w
pep.Thanks all
Marandahas left
Marandahas joined
MattJThanks pep. :)
moparisthebest> One doesn't have to use all the budget anyway :)
maybe you should talk to the government
Wojtekhas joined
adiaholic_has left
adiaholic_has joined
Jeybehas left
eevvoorhas left
GuusI blame Frozen. She couldn't get out of her princess dress fast enough. Second time I had to clean the restroom today. The other kid was sick in it earlier.
GuusMattJ: thanks for taking care of things. I'm not exactly sure on what happened in the first place. Would you mind sharing some details when you have some time?
MattJWe have no details :(
MattJI don't even know what time the server went down
GuusSo, we 'just' have an unresponsive server and no way to get to it?
MattJYes
MattJFor what it's worth I found in my email archive that this also happened in March 2019
MattJAnd a power cycle brought it back to life
MattJBut the guy who did that last year has not been responsive so far (and many DCs are locked down right now anyway)
lovetoxhas joined
GuusTo what extent do we have backups from which we can restore data, if need be?
MattJYou're currently chatting on a server restored from backups
calvinhas joined
adiaholic_has left
MattJThe backups are nightly, so we potentially lost up to 24h of chats/wiki edits, depending on exactly when it went down
GuusMail archives?
MattJMail is on a different server
MattJ(and that is also backed up)
GuusIn the same data center though?
MattJYes
GuusSo the data center didn't explode. 😁
MattJYeah, other stuff is working fine there (and the other servers are in the same rack)
KevFrankly, I think it's a wonder that Matt has managed to get 'everything' back up from a backup, given backups invariably turn out to not work when you need them. So despite being an inconvenience, I think this is a win!
MattJYeah, this could have been a different story :) (one we've lived through before)
GuusHehe, glass half full kind of guy you are.
MattJSo thanks Kev for leading that initiative back then
MattJAlso, it's not necessarily the case that data on the down server is lost
MattJJust unreachable until it's powered back on
KevAlthough at least for wiki edits it effectively is.
MattJI don't plan to merge anything if it comes back, if that's what you mean
KevBecause chances of us mananging to work out and reapply deltas between two DBs are limited.
KevChat logs for a single day are probably easier to reinsert.
MattJBut I wouldn't mind checking 'Recent changes' on the old server and comparing
KevYeah, that wouldn't be daft.
MattJand a manual copy/paste in that case would suffice, likely
eevvoorhas joined
GuusThanks for taking care of everything. Let's find a way to prevent this from happening again in the future. If budget is needed for that, I think we should consider spending that.
debaclehas left
MattJIt's a difficult situation - if we can continue to use donated hosting, I'm totally fine with that
ZashGiven that there were days between the crash and there store it should be relatively easy to merge the chat logs
ZashSays me with my Prosody hat on, knowing how that's stored.
Zash(hats!)
MattJWe just need to ensure that things like this don't happen, and that we have an action plan when they do
KevI think I suggested something about this? Or did I dream that conversation?
GuusObviously, but I'd like to have a reasonable assurance that we can reach our servers when they're out. Last time, this was an issue too.
MattJI'm pretty sure if the XSF wanted to e.g. pay for a power strip that allows remote power cycling, that's already an improvement
MattJKev, you did say similar in a private conversation with me
pep.MattJ, I doubt you'd need this tbh
pep.IPMI/ME should do, if available
pep.and I guess that is, on a server
pep.Now it does need to be plugged :p
MattJI /think/ the server in question doesn't have that, but I may be wrong
alexishas left
MattJSome of them do, but it's not exactly documented (except in ancient email threads)
alexishas joined
KevTypically we've bought hardware that does have ILO-ish things, but this box was different so I don't know.
GuusThere's probably a bunch of options that we can look at. I'm happy for iteam to first finish the disaster recovery, then come up with a plan for the future.
GuusWhat's confusing to me is that in a DC, we're still limited to one contact person.
GuusIsnt this a commercial DC, or are we serviced off the books or something?
MattJOur relationship with them pre-dates my involvement with the XSF, so I don't know the details
KevThis is all done out of the goodness of Jerry's heart, and has been since the year Dot. We're not a client or anything.
GuusAlthough I'm very happy for Jerry's efforts, some kind of more robust service might be preferable, given that our hardware seems to become less reliable.
GuusI'm thinking we are running on aging hardware?
KevThat's also because we live off donated and/or ancient hardware etc., mind.
GuusSure, it is totally understandable
GuusWe have a pretty good deal on all this
KevI remember exactly one issue with the hosting, as opposed to the machines -and far more instances of the opposite, where Jerry's just found 'random' hard drives/other hardware to patch up our servers to get them straight back up.
Kev(And that issue with the hosting was because we were getting DoSd so Jerry had to DoS protection stuff)
MattJAnother part of me thinks that we don't /need/ dedicated hardware for what we do these days. Maybe in 2004, but virtual servers are cheap and easy these days.
GuusBut maybe it's time to improve reliability by spending a bit more money
KevSo I'm very happy to recommend various changes, but I don't think our hosting environment is a significant part of our issues.
archas left
archas joined
KevYes, we could probably (other than xmpp.net) cope with a £5/mo servers4u VPS or something.
GuusIf an important server is unreachable for several days, I think it is somewhat of a problem.
pep.MattJ, depends what we want to do with them
pdurbinhas joined
MattJIt's been a week now
MattJ(well, it went offline some time between Thu night and Fri morning)
archas left
archas joined
lovetoxhas left
pdurbinhas left
archas left
archas joined
archas left
archas joined
krauqhas left
krauqhas joined
sonnyhas left
eevvoorhas left
werdanhas joined
adiaholic_has joined
lovetoxhas joined
j.rhas left
j.rhas joined
edhelashas left
edhelashas joined
Jeybehas joined
adiaholic_has left
adiaholic_has joined
mukt2has joined
mukt2has left
goffihas left
Steve Killehas left
goffihas joined
Steve Killehas joined
etahas left
etahas joined
archas left
LNJhas left
archas joined
archas left
archas joined
Yagizahas left
LNJhas joined
eevvoorhas joined
sonnyhas joined
João Duartehas joined
waqashas joined
alexishas left
Jeybehas left
Jeybehas joined
Steve Killehas left
mukt2has joined
Jeybehas left
Robertohas left
Steve Killehas joined
Tobiashas left
Tobiashas joined
archas left
archas joined
Alexhas left
Alexhas joined
archas left
archas joined
João DuarteI just realized now, after 4 days, that my clients werent updating any messages
Half-Shothas left
Half-Shothas joined
Alexhas left
Alexhas joined
João DuarteWas coming here to give Manav some feedback, but now if i understood correctly, it was some server issue that is causing trouble to the chat rooms?
mukt2has left
flowJoão Duarte, yep
João DuarteAh okay. Anyway to get things back updating in the clients? Or just wait?
ZashDepends on the client, but leaving and rejoining should do the trick
Alexhas left
Alexhas joined
João DuarteIm using pidgin on desktop and xabber on mobile
João DuarteAlready tried rejoining to no avail
João DuarteBut maybe i can troubleshoot my way through, just wanted to check it was within my reach and not something purely on the server
ZashThe server that served this chat went down and we don't know why. This chat is now running on a new server.
João DuarteSame thing for prosody's chat?
ZashNo
Alexhas left
Alexhas joined
João DuarteI guess so, because the last messages I have were sent in the 22th
João Duartehm
ZashThen it was probably not the XSF server issue.
Link Mauvejonas’, you once told me not to include real client names in XEPs right? I’m fixing MIX’s examples which were violating XEP-0030 MUSTs.
Link MauveAh no, in the end you accepted it in XEP-0402.
Alexhas left
Alexhas joined
adiaholic_has left
adiaholic_has joined
João Duartehas left
edhelashas left
edhelashas joined
Jeybehas joined
Nekithas left
Jeybehas left
Jeybehas joined
remkohas joined
Link Mauve“A nick MAY be specified as a <nick/> child elements of the <join/> element.”, what happens if the user’s server didn’t specify a nick and the MIX channel requires one?
Ge0rGLink Mauve: yes.
Wojtekhas left
Nekithas joined
Link MauveAh, “If nick is missing on a channel where nick is mandatory, the join MUST be rejected.”, nvm.
Wojtekhas joined
Robertohas joined
Jeybehas left
adiaholic_has left
mukt2has joined
pdurbinhas joined
Link MauveAgain in MIX, why is it <update-subscription/> but <setnick/>?
eevvoorhas left
eevvoorhas joined
mukt2has left
werdanhas left
Link MauveIn section 7.3.4, this is the first time there is no <{mix}destroy/> in the iq result, is this wanted?