XSF Discussion - 2020-08-25

Well technically it's probably incitement. Which is illegal. But INAL and nobody is going to sue over that.

lovetox, You're not allowed to send data to the FBI; it would be illegal under the GDPR.

moparisthebest, But, a GDPR Subject Access Request over XMPP would be legit and require a response within a fixed time period of a month (and without "undue delay").

moparisthebest, (This case isn't a DSAR, of course, as the FBI is not a Data Subject here)/

Memberbot is still online for our member meeting today. If you are a XSF member and have not voted yet then please do so. Thanks.

Alex, Thanks! I won’t be able to attend in-person, but I already voted :)

Just voted :)

btw, I've been thinking about ways to solve s2s connection issues (and push notifications) with MUC/XEP-0045 in ways that are mostly invisible to the clients participating in the MUC. I wrote up a 1,900 word very rough form of a XEP, and will probably look at converting that into a "proper" ProtoXEP soon :)

the basic idea is to have the user's server perform MUC Self-Ping on the users' behalf and intelligently handle rejoining & history replay for the client, removing the need for the client to support such logic

MUC servers must implement Unique and Stable Stanza IDs for the catchup to work, and are strongly advised to implement MAM and the optimization parts of MUC Self-Ping (but these aren't hard requirements)

eta, that doesn’t solve all of the issues

unfortunately

eta: is that prosody mod_minimix ?

but it improves the situation

jonas’, do tell

what issues remain?

moparisthebest, no, it's something different

eta, ah, well, if you force servers to do MAM catchup and replay on behalf of their users, that’s good

otherwise this won’t quite work

jonas’, wanna write down all the issues on the wiki or such?

Zash, -EBUSY

-WSADFACE

jonas’, so how I've spec'd it is that if your client conforms to this new XEP it just gets a "room oops happened, please do MAM resync" message and handles MAM itself

I just came back from the police department (which was already closed) and that kind of stole more of my time than I had budgeted for

if your client doesn't advertise support, the server will do MAM for you and forward the messages

eta, ah, that’s not at all "invisible to clients" then

and not much of an improvement over self-ping, is it?

why not always let the server do MAM?

jonas’, because paging

I don’t follow ✏

let's say you're disconnected for an extended period and miss out on like 1000 lines of history

yeah

now the server will perform MAM to filter through that to see if any of those lines contain your nickname (for push purposes)

but it won't want to buffer all 1000 and send them to you, because that drains the server's resources unnecessarily

but if a client doesn’t support it, the server still has to do it, right?

yeah, so there's a limit

I don’t like that

don’t hide issues if you can’t hide them completely

Will perform MAM ... when?

Periodically?

Zash, after MUC self-ping errored

Zash, upon reconnect

Polling? Hmmmm, not sure if like

Zash, polling is required

ok, well

"Fundamentally, due to the way s2s links work (i.e. they get suspended on lack of activity), we'll always need to perform some kind of polling to make sure we're still in a MUC. Doing things like the server proactively saying it's back and clients should rejoin is *good*, but it'll never cover all cases due to the nature of networking. Therefore, it's probably a good idea to have a user-configurable polling interval that defaults to something like "10 minutes since last activity observed to/from the MUC" that people can tweak if they really care about knowing about disruption earlier."

I shall write stuff down later

Zash, also this is a strict improvement upon the status quo, right

no

clients currently have to poll anyways

it's better if the server does it once than having all connected resources do it

if you hide message loss (the > 1000 messages situation), that’s not an improvement

jonas’, it's not hidden

how is it not hidden?

jonas’, you either support the XEP, in which case you get an explicit "do a MAM resync" notification, or you don't, in which case you just get kicked

if the client does not advertise support, servers will simply not send all of the history, but fake as if the client had been in the room all the time.

no, they won't

it kicks you in that case

hue?

what about the MAM stuff you said earlier then?

so it tries to do a catchup

ah, and if that’s too much, you get kicked

makes sense

yeah

and then you just do whatever retry logic you currently do

(except it'll tell you that it kicked you for this reason, so you can instarejoin knowing that you're actually still in the MUC and just need to do MAM)

current retry logic: none

Count me as interested, I've long thought we need to fix this on the server side, but haven't been able to put in the time to spec anything up

I mean I'll write this up into a proper protoXEP at some point soon once I figure out how to do that

the current doc is somewhat incoherent

Step 1: obtain markdown-to-xep ✏

if there are no glaring issues with it, I'll probably forge ahead with trying to write prosody modules

oh yeah, the other thing I forgot to mention is

eta, ok, that makes much more sense.

this XEP has bookmarks2 as a hard dependency

:)

eta, I’ll be happy to help you through the XEP process -- let me know if you need any technical help in getting this in the right format

speaking of right format: MattJ, do you have it on your todo list to clean up the new mam-forked XEPs?

basically the flow is without a bookmarks2 entry for a room, you don't get any server fanciness

jonas’, what needs cleanup specifically?

so to engage the fanciness, you join normally, then add a bookmarks2 entry

MattJ, they are lacking some RECOMMENDED/REQUIRED sections IIRC

(at which point the fun state machine specified in the XEP kicks into action and joins another resource that the server controls)

Ok, that wasn't on my todo but I can look into it

Probably not for a couple of weeks though

MattJ, I see, thanks nontheless

nothing too urgent anyways

once the fun durable mode is engaged, you never get kicked out of the MUC unless (a) someone actually /kick-ed you or (b) the server gave up retrying to get you back in (usually after like a day) ✏

in both cases, something is shoved in the bookmarks2 entry to say "this MUC is broken", which disengages the fanciness until you remove that by manually rejoining and re-adding the bookmarks2 entry

eta, I’m sure I can read that in the document later on, but what happens if the user sends a message while they are not fully connected?

jonas’, what do you mean by "not fully connected"

eta, the server is currently in the process of retrying to get you connected to the MUC ✏

jonas’, you get an error of type "wait"

the one smoking gun might be if current clients decide to rejoin the MUC in this case, which would suck a lot

well, actually, no it really wouldn't ✏

that depends on the specific condition, but a server could easily eat rejoins away in such cases.

jonas’, it kinda does

if you attempt to rejoin whilst in the `Reconnecting` state, what happens is it marks you as "waiting to rejoin", eats your presence, and sends you a "experiencing connection issues" message (for conformant clients)

if it manages to get you to join, it forwards you the recipient list and then does the "intelligent catchup" process

actually wait, no, it wouldn't do that because you'd be expected to do MAM anyway, so scratch that

Bonus points for a state diagram :)

https://theta.eu.org/xmpp/upload/3OHj4KIs8L9tD3JW/7543125f-08d5-411e-9732-722daa038bd5.png

MattJ, 👀️

Bonus points awarded, achievement unlocked!

the nice thing is, if this XEP actually becomes a thing the UX will probably be better than something like matrix

because you'll get an explicit "experiencing connection issues" display in your client's UI

(which I think is a lot nicer than just sending and not getting any messages back until s2s reconnects and everything floods back in)

also, there's no requirement to implement a bloody complicated MIX server

like all you need on the conference service side of things is MUC, unique and stable stanza IDs, (MUC Self-Ping optimization, MAM)

it makes MUC better, but it doesn’t solve other problems of MUC ;)

so I can write my transports and have them work :p

in fact, this is something MIX also still needs to solve, so I think that work is valuable

jonas’, what are the other problems though?!

oh, MIX

nvm

no, your proposed thing

okay, what other problems are there

try to address a specific resource of a participant in a MUC

what's the usecase for that?

the entire multi-session nick hack which isn’t written down anywhere normative

eta, sending IQs

ah

oh yeah, re nicks, you can't specify one on join any more

it forces it to be your bookmarks2 one if you have one in bookmarks2

I need to step away for a bit, I’m having a hard time to concentrate on anything

sure

I take your point that there are other problems with MUC though :p

(but there can be more specs for that)

there's always other problems with MUC :p

What do we need muc for anyway?

Are you going to say... XEP-0033? :)

YES

For smaller groups I can certainly see the appeal

Would be interesting to see that

chatrooms without MUC 🤔

it's a bit the difference between mailing lists and chain-answered-emails

exactly that

a, somewhat major server implemented XEP-0033 recently (jmp.chat/cheogram.com) but no support in Conversations (or most/all clients?) yet

ah yes

group emails

that very functional solution that never results in people getting confused ever

can't wait for XMPP Chatrooms over SMTP

From 0313's changelog: > Split preferences protocol into a separate document Which document did they go into?

Holger: New ones, via the inbox. (Council voted it)

Ah, thx.

Dunno if Editor has processed them yet

they haven’t been voted into existence yet

Zash is on-list on them

Oh right

*hinthint* if you vote properly within the next few minutes, they might make today’s editor window *hinthint*

eta - out of curiosity - why not push for MIX addotion, instaed of putting another patch on a MUC?

Wojtek, I think MIX is overengineered

and yet another patch on (kinda broken) MUC is better?

only one can be used without all servers upgrading at once

^

MIX isn't backwards-compatible with MUC

MUC is a very simple standard, and I like that

Wojtek, MIX also has exactly the same issue

(not quite exactly, but well)

eta, MIX does have a MUC compatibility layer which I’d deem "good enough" for a soft transition

jonas’, okay, but also

eta: XEP-0408?

MIX creates an ecosystem split where half the users still have a crap experience

in tigase's defense they *actually* put in the work and seem to have made a seemingly working implementation, I'll have to update https://www.moparisthebest.com/mix/

even with a compatibility layer

eta, that’s life.

jonas’, no, but it doesn't have to be this way

eta, yes, hence I’d actually suggest putting energy into MIX -- but I don’t blame *you*, because the work you’re doing we need for MIX, too

we discussed about that in this very room just the other day

yeah, I saw :)

Wojtek, okay, point taken

eta, MUC may be a simple standard yet time and time again there are problems with it and (weird ways) to fix that...

MUC is by no means a simple standard

@jonas’ what same issue?

Wojtek, dealing with s2s outages

clients need to know about it to sync their archive with the remote MAM (if we go for remote-archives-only) and/or servers need to sync the user’s MAM and do magic if we allow for user-local archives

eta, MUC is simple? try to make it scalable, consistent, etc. you will learn that it is not so simple

it's simple if you're a transport developer :>

eta, MUC is a very long document as it stands already. Aside from that, there are hacks upon hacks which aren’t even *documented* there (like the whole multi-session nick stuff, or IQ routing in MUC in general)

@jonas’ if we *don't* allow you mean? beacause with local archive each time there is a new message it's delivered to local archive which eliminates issue with s2s outages

and MIX solves this problem?

Wojtek, no, if we *do* allow.

Wojtek, that’s not correct. what if the s2s between MIX and user server is down?

eta, yes, because MSN (multi-session nicks) as well as IQ routing (by having each resource addressable) are both well-defined in MIX

then your local archive won't get the message, but you also won't be able to query it ;-)

Wojtek, yeah, so you have message loss (bad)

but I get it now

still, I don't recall fallback to query remote server if `urn:xmpp:mix:pam:2#archive` was advertised

it’s a rather tricky thing to solve. It’s not just that servers do MAM syncs among each other. To ensure correct order of message delivery, servers will also have to do replay from MAM as well as queue "live" messages

Wojtek, sorry, I lack context, what does `urn:xmpp:mix:pam:2#archive` mean and who would fallback to what?

"Archive of MIX channel messages MAY be performed by the participant's server. When this is done, the capability is advertized to MIX clients using the 'urn:xmpp:mix:pam:2#archive' feature. If archive is provided it **MUST always be used**, so that where a message is sent to the participant's server and discarded because there are no active clients, it will still be archived. This means that when archiving is provided, the messages will be available in the local archive and can be picked up by clients when they come online. "

@jonas’ simples solution for s2s issues would be stream management over s2s, or force MIX to add stable-id of a last message which was sent, so that users server MAM could fill that and deliver "lost" messages as a normal MIX message with `<delay/>` element in them

Andrzej, my proposed solution does the stable-id thing, sorta

Andrzej, no, stream-management is not sufficient

stream management will have limits because servers won’t queue forever.

Wojtek, the text you quoted is exactly the problematic part. It claims that the user’s archive is the gold standard, but in fact it’ll have gaps without additional measures.

Andrzej, what you propose is one of the ways to do it, but then the user server becomes quite complex. It needs to do MAM queries, backfill the local archive (which it may not be able to even to, because it’d have to insert into MAM at an earlier point, violating one of the MAM guarantees!), it has to buffer all "live" messages from the MIX to be able to replay MAM first in order to provide in-order delivery of all stanzas from the MIX.

and it needs to do that across all pubsub-nodes the user has subscribed to on the MIX, too

@jonas’ we will have gaps, but it was discussed at the summit and the simples solution to add to the message stable-id of a previous message was suggested (by Kev if I recall)

and then the client would be responsible for filling the gap in its local archive?

@jonas’ why it needs to insert those new entries in the past? just add them at the end with `<delay/>` element in them

then MAM would be just log of a messages which you have received and delay timestamps would allow clients to assign those messages in proper order

which clients don’t do generally

and I’m also not sure if that’d be acceptable

Wojtek, XEP-0408 doesn't really say much

I know that it's the problem, but it says that if there is local archive the user *MUST* use it (so no queryign from MIX MAM)

Wojtek, hence the debate about that particular text :)

eta, needs implementations. but conceptually, it can easily work.

okay, so MIX removes some hacks that you need to make MUC work

I mean I don't think the problems with MUC are bad enough to justify scrapping it entirely

or rather, to justify the ecosystem split having 2 competing standards would cause

that is your opinion and you can have it :)

:-)

indeed!

if people want to implement MIX though, go ahead! part of why XMPP is great is precisely that people can have different ideas

and having many things of doing something is kinda "XMPP way" ;-)

eta, for mobile devices (ie. iOS) using MUC requires a lot of hacks

though I *do* find it amusing, that the often-complained-about required user-server support for MIX is now coming for MUC ;)

for me it seems that MUC is basically mutating into MIX, in stages...

Andrzej, are a lot of hacks bad though? :p

YES :-P

Andrzej, I mean the followup XEP I was going to write is a spec for doing push notifications

if the server has a resource constantly joined to the MUC, it can receive messages, match them against a set of rules / regexes, and then forward the message to the client

eta, with filtering I hope?

eta, what about E2EE?

Andrzej, yeah, I want to essentially rip off matrix's push rules system

jonas’, you'd have to either (a) send every message or (b) include a separate <mentions/> element or w/e

and deal with the privacy implications of (b)

> if the server has a resource constantly joined to the MUC, it can receive messages, match them against a set of rules / regexes, and then forward the message to the client so, basically permanent member subscription? like in... MIX? ;-)

although *no* messenger can deal with the e2ee problem :)

eta, if you want to write something like that please check https://xeps.tigase.net//docs/push-notifications/filters as this could be useful

Andrzej, issue I have with that is, what does "mentioned" mean

do I get a notification any time someone says "details"? ;P

it means you can silently eat the mobile battery of everyone you are in a channel with of course :)

mentioned means that your nickname was mentioned in a message

Andrzej, no, sure, but how are you matching

if your test is `string.contains(nick)` it's going to break for me :)

contains and check if before and after there is no alphanumeric character

ah, good

that aside, though, yeah, that XEP looks very similar

Wojtek, yeah, exactly like in MIX :p

I was suggesting to consider just requirements from our XEP (and syntax if that would be useful)

like I don't think MUC's current semantics are ideal at all

Wojtek, the difference is my proposed XEP doesn't require the MUC service to support much more

but requires local server to keep a persistent session for you

yeah

with S2S issues, reconnections, MUC restarts, etc. this can be problematic

Andrzej, that's what the whole XEP is about

you could even extend it to have local archiving semantics similar to your MIX implementation

it just won't do that initially because that's a bit of a hard sell for some server operators

generally my problem with MIX is it doesn't really obey Postel's Law (https://en.wikipedia.org/wiki/Robustness_principle): it requires there to be a shiny new MIX service out there to get any of the benefits

whereas trying to patch MUC on the server side will improve experience for ~90% of users without them having to change client or MUC service

that said, for non federating environments, MIX is unquestionably better

and then you end up in a situaiton when some server support that and some not and some user will have different experience than others... and then everyone complain again that it's impossible to have more-or-less consisten experience with XMPP and jump on the matrix badwagon

not sure that's the right law, it requires a flag-day, which never ends up happening

https://en.wikipedia.org/wiki/Flag_day_(computing)

Wojtek: well no, you make it part of the compliance tester

also I mean, MIX has the same issue!

(in fact it's worse)

huh TIL that flag day was also Postel's https://tools.ietf.org/html/rfc801 :)

compliance server is run by a private person and not XSF… and it's kinda arbitrary - it requires XEP-0215 yet compliance suite doesn't mention this particular XEP...

it also doesn’t say XMPP or XSF compliance ;)

The compliance tester is just 4 month ahead of time

Wojtek: well, I mean I'd be interested to hear your solution here

fragmentation is definitely an issue, but I'm not sure there is an obvious magic bullet

eta, the idea to make it dependend on bookmarks2 seems like a sure way implementation gets delayed very long

there is no server that supports bookmarks2 in a way that client would want to use it

lovetox: but bookmarks2 defines a private XML conversion path

lovetox: oh? what are the issues with bookmarks2?

hm? i didnt say there where issues with bookmarks2, only that no server supports it

ah

and bookmarks2 in turn depends on other XEPs that servers dont support

does it?

like the recent "max" change in pubsub

> it also doesn’t say XMPP or XSF compliance ;) yet it seems that everyone considers it as such ;-) and eta proposed using it as a mean to promote XEPs :-)

> The compliance tester is just 4 month ahead of time hence "arbitrary" - you can do with compliance tester whatever you want as it's not XMPP/XSF :-)

Wojtek: hey, feel free to make a Siskin compliance tester :p

Siskin is client, not server ;-)

Wojtek: no, but the other tester is "the Conversations compliance tester", because it tests whether the server works well with Conversations

"A thing that tests whether servers are tuned for Siskin"

Wojtek: like, more compliance testers would be great!

I'm not saying Daniel's one has to be the One True Tester

eta yet you proposed to promote certain specification via compliance tester

so is it really a "compliance tester" or "compatibility tester"?

Wojtek: well, okay, that's assuming Conversations uses said specification

Andrzej: I was just going to say ... the later.

Wojtek: perhaps I'm a bit too optimistic in that regard :p

You could even fork it, tune it to your own clients requirements.

Or make an ultimate every client compatibility tester

Wojtek: my point is just that the conversations compliance tester seemed to do a relatively okay job of driving AV call adoption by servers

(along with a client needing it and users wanting the call feature)

true :p

> element header: validity error : Element header content does not follow the DTD, expecting (title , abstract , legal , number , status , lastcall* , interim* , type , sig , approver* , dependencies , supersedes , supersededby , shortname , schemaloc* , registry? , discuss? , expires? , author+ , revision+ , councilnote?), got (title abstract legal number status type sig approver dependencies supersedes supersededby author shortname revision ) </header>

oh my god

they have to be in the correct order

that's ridiculous

Praise schema validation!

eta, I recommend using xep-template.xml

jonas’, ...now you tell me

I told you to ask :D

I've already written an intro, which is good

eta, next time, ping me directly, I’m happy to help before things become an annoyance

not having to refer to XEP-0001 all the damn time is nice though :)

But have you considered implementing an elaborate document processing pipeline that turns plain text into xep-xml?!

GPT-3?

Oh no

hmmm

*spits out machine generated XEP* hmm looks alot like matrix

You're holding it upsidedown?

I turned it upside down and now it just looks like a blockchain

let’s see if they give it to me

lets kick off our member meeting

!

\o/

oh right! and I’m even around!

here is our Agenda for today: https://wiki.xmpp.org/web/Meeting-Minutes-2020-08-25

1) Call for Quorum

as you can see 34 members voted via Memberbot, so we have a quorum

2) Items Subject to a Vote

new and returning members, you can see all applicants on this page: https://wiki.xmpp.org/web/Membership_Applications_Q3_2020

3) Opportunity for XSF Members to Vote in the Meeting

anyone here who has not voted yet and wants to do so now?

looks like none

I will shutdown the bot then and start working on the results

🥁

4) Announcement of Voting Results

when you reload the page at: https://wiki.xmpp.org/web/Meeting-Minutes-2020-08-25#Announcement_of_Voting_Results you can see the results

all new and returning members are accepted. Congrats to everone, and welcome to our new members

5) ny Other Business?

None from me

None here

except: congratulations to everyone and thanks to our Secretary for the duty

6) Formal Adjournment

I motion that we adjourn

+1

thanks everyone

Thanks Alex !

🔨️ thanks Alex

Thanks Alex. Congrats to all.

will update the memberlist and mailing list tomorrow morning

Thanks to Dave for updating Memberbot. Was running very smoothly this time ;-)

Special congrats to (!)!XSF_Martin and Holger :)

Thanks 😃

The irony is that the original "XSF Martin" that caused you to switch to that nick... I'm pretty sure he was never a member of the XSF :)

not?

but he was in board, right?

jonas’: yes

ok, that explains my assumptions (and I suppose also mdosch’s)

how do clients typically advertise support for some XEP?

eta, XEP-0030

jonas’, right, so the server is expected to do the whole entity caps dance and disco#info query it?

yes, servers do that anyways tho

for PEP

oh, okay, so that's something we can build on

so just state that the client has to implement XEP-0115 and go ahead

right, after 20 minutes of XML wrangling I've managed to make a TOC

actual spec content still to follow >_>

(well, there's an intro, requirements, and MUC failure modes section)