XSF Discussion - 2021-04-13


  1. jonas’

    fg

  2. MattJ

    Hey!

  3. dwd

    Ge0rG, It's looking very like I'll have copious amounts of free time quite shortly, and if so I'll happily help out with performance work etc on Yax.im.

  4. Ge0rG

    dwd: are you really sure about that? This is a deep hole full of unshaved yaks!

  5. mathieui

    for the yak horde!

  6. dwd

    Ge0rG, In fact, I can probably help anyway, but I'd need some access to a pgsql instance if you're switching to that. I got ours to cope with 700% increase during the pandemic. It was... fun?

  7. Ge0rG

    dwd: TL;DR: my prosody+sqlite combo is running into a limit of ~200 messages stored into MAM per second, causing starvation of other connections

  8. dwd

    Ge0rG, Yeah, we have PostgreSQL handling about 1.3 million per hour at peak. I need to do that talk, don't I?

  9. Ge0rG

    dwd: I've done some first synthetic benchmarks of psql on the yax.im machine and it reduced per-message storage processing from 3.5ms to ~1ms

  10. dwd

    Ge0rG, Where are you hosted?

  11. Zash

    dwd, is that 360/s ?

  12. Ge0rG

    dwd: in Berlin, at my ISP's uplink provider, running some LXC in KVM on baremetal

  13. dwd

    Zash, Yeah, but it's doing realtime analytics as it goes, and encrypting the message content in the database for regulatory reasons, which is the slowdown. It does actually get faster than that, perhaps 500/s.

  14. dwd

    Ge0rG, So "hand installed" PostgreSQL, not something like AWS RDS.

  15. Ge0rG

    dwd: from my first synthetic stats, I might get from 200msg/s to something around 300-500 by switching to psql, which might or rather might not suffice to cope with the VaxBot data flood

  16. Ge0rG

    dwd: yeah.

  17. dwd

    Zash, Oh, and full-text indexing. Yeah, that costs, too.

  18. Ge0rG

    this is my schema: https://pastebin.ubuntu.com/p/sC5H7GRF9Q/

  19. dwd

    Ge0rG, Do you have pg_stat_statements available? That should tell you where the costs are. Also you don't need both the "prosody_index" and "prosodyarchive_index", the latter includes the former. Beyond that, I'd need to understand what the SQL queries are, but I can find that from the source I suppose?

  20. Zash

    dwd, those are different tables tho

  21. dwd

    Ge0rG, (Also I may have to track down and murder whoever wrote that schema using reserved words as column names)

  22. Zash

    Just quote your friggin column names

  23. dwd

    Zash, Oh, indeed. Is that a generic keystore in SQL?

  24. Zash

    key-value store, yes

  25. dwd

    Zash, Well, I'll have to thanks to the dumb naming. :-)

  26. Ge0rG

    dwd: the queries are done through two or so abstraction layers :(

  27. Ge0rG

    dwd: did I mention I'm still on sqlite?

  28. dwd

    Ge0rG, OK, so for postgreSQL, you'll want this: https://www.postgresql.org/docs/current/pgstatstatements.html

  29. Ge0rG

    dwd: noted

  30. dwd

    Ge0rG, And in a perfect world, we'd do internal observability with something like New Relic. Oh for a server not written in a novelty language! ;-)

  31. Ge0rG

    Well, I'm probably also running into a CPU bottleneck, either from creating debug logs or from memory management

  32. MattJ

    New Relic = $$$$

  33. dwd

    MattJ, Yes, but amazing, and their new pricing works out close to free for hobby projects.

  34. dwd

    MattJ, I believe you can get some of the benefits from Prometheus and similar, but I've not seen it done.

  35. emus

    Nice, F-droid is following XMPP on Fosstodon. Hopefully, they boost one or the other toot đź‘Ť

  36. dwd

    The guy who started F-droid, Ciaran, ran his own server for years though.

  37. dwd

    Lovely chap.

  38. emus

    ah okay nice + we have 1200 followers now. thats 400+ in the last two months I think

  39. Zash

    Graphs? Graphs!

  40. moparisthebest

    I think the one-two punch of whatsapp screwing their users and signal becoming a shitcoin wallet really helped

  41. dwd

    Shame so many XMPP jobs are in the funny-money business these days.

  42. dwd

    Though I admit I've never quite understood the connection.

  43. Kev

    Blockchain is trendy, XMPP is trendy, two great tastes that taste great together.

  44. Kev

    Blockchain is trendy, XML is trendy, two great tastes that taste great together.

  45. dwd

    XMPP isn't trendy, it's just riding the XML craze?

  46. Kev

    Right.

  47. Ge0rG

    All my appointments are getting delayed, but I'd like to come to office hours

  48. emus

    https://fosstodon.org/@xmpp/106058677871732292

  49. dwd

    Well, that was fun and interesting, thanks Sam!

  50. Zash

    👍️

  51. emus

    Yes, I can agree to that. I think you hit a sweet spot, also with the XMPP Hours in general!

  52. Sam

    Thanks for coming!

  53. Sam

    I was worried about that one since we hadn't tried a roundtable, but I'm happy with it.

  54. Zash

    Maybe you should force us to introduce ourselves, so we get used to the concept of talking into the mic 🙂

  55. Zash

    Also need to figure out when to make dinner in relation to these slots.

  56. Sam

    Good idea.

  57. Sam

    Sorry about the food time overlap; this slot isn't exactly ideal but the most people marked it as green

  58. Sam

    Threw the notes up, sorry that the formatting didn't translate: https://wiki.xmpp.org/web/XMPP_Office_Hours/2021-04-13-Notes

  59. jonas’

    Kev, friendly ping https://github.com/xsf/xeps/pull/1047

  60. Sam

    dwd: Unsure if you were still there when I said this, but if you need to do your talk "soon", feel free to bump mine next week. But if that's too soon, there should be plenty of weeks open after the week after that

  61. mathieui

    Thanks for the notes Sam, I haven’t yet been able to attend office hours, but it is great!

  62. Sam

    I just realized I'm getting my second dose of the Pfizer vaccine right before office hours next week, so we'll see if I'm too busy creating antibodies and am hurting or falling asleep :) I have friends who said the second shot really knocked them on their ass for a day or two

  63. Sam

    But I don't remember what manufacturers shot that was, so maybe I'll be loopy or something during the presentation :)

  64. jonas’

    you should be adding a trigger warning to such statements because they can easily cause envy

  65. Daniel

    i was about to say the same

  66. menel

    Sam: that takes a few hours to sink in so you will be fine for just until the talk is over :-)

  67. Sam

    ooh yah, sorry

  68. mathieui looks at his 2024 vaccination date

  69. jonas’

    (I wasn’t really serious)

  70. jonas’

    I’m happy for all the folks who can get protection already

  71. Sam

    I love mastodons content warning feature, we need that for xmpp :)

  72. jonas’

    spoilers?

  73. mathieui

    Sam, we have spoiler messages already !

  74. menel

    Xep-spoiler warning for conversations

  75. Sam

    We need that to be supported in XMPP then

  76. Zash

    Converse.js supports it AFAIK

  77. Sam

    I thought it had limited usage until I joined mastodon, now I love it for anything longer than like a sentence or anything I'm just tired of seeing and assume others will be too

  78. deuill

    Awesome work with note-taking.

  79. Sam

    I have no idea who was doing that, but it wasn't me after the first few minutes; thanks to whomever silently took over while I was struggling to both call on people and pay attention and note take

  80. Zash

    further evidence that chair and minute taker make sense as separate roles

  81. jonas’

    are the notes funny? if yes, it was Tedd

  82. mathieui

    sorry, reading the minutes for the office hours and I don’t know if "compliance suits" is intentional or not but it is pretty funny

  83. Ge0rG

    mathieui: are those orange?

  84. mathieui

    don’t ask me, I haven’t read that XEP yet

  85. Ge0rG

    Probably it means the white ones, with the long arms that you bind behind the back

  86. mathieui

    I only have like 5 XMPP-related pieces of clothing, none of which are suits!

  87. Zash

    Suits instead of hoodies for the post-pandemic Summit/FOSDEM?

  88. vanitasvitae

    +1

  89. vanitasvitae

    or snobby bathrobes!

  90. Zash

    Doesn't the logo sorta look like an up-side-down untied tie?

  91. Ge0rG

    https://upload.yax.im/upload/GbsmIjpFUgjFP3q4/hmgoepprod.jpg

  92. Ge0rG

    Here's Bruno branded

  93. dwd

    XMPP dressing gowns? Or xmpp smoking jackets?

  94. Ge0rG

    XMPP straitjackets

  95. Ge0rG

    (that's at least my association with "compliance suit")

  96. moparisthebest

    Sam, OT but my second phizer shot gave me the same side-effects as the first, felt like I'd been punched in the arm a lot, that's all, YMMV of course :)

  97. Sam

    I had nothing from the first one really, just like getting a flu shot or something, tiny bit of stiffness but nothing really

  98. moparisthebest

    did anyone play that game in school where you and a friend would take turns punching each other's arm as hard as you could until one of you gave up? that's what it felt like lol

  99. Ge0rG

    Those are very strange games.

  100. Ge0rG is playing the game of xmpp server meltdown to help another country get vaccinated despite that country prohibiting vaccine export to where the server lives

  101. moparisthebest

    now *that's* a strange game

  102. mathieui

    which reminds me I wanted to setup a quick & dirty french vacbot

  103. mathieui

    happy to get jabberfr to meltdown as well for this fun game

  104. mathieui

    or should I recommend yax.im?

  105. Ge0rG

    mathieui: easy. `onMessage() { return "sometime in 2024"; }`

  106. jonas’

    mathieui, decentralization!

  107. mathieui

    Ge0rG, nah, for people above 55 it should be quite doable

  108. deuill

    This Office Hours session will probably be the starting point for a few more, the way discussions went. It's awesome to see people involved in the project in very different ways providing their views, and where these overlap or diverge.

  109. Sam

    Only if people actually think about what they want to discuss and put it on the calendar :)

  110. deuill

    Some threads I'm picking up: - Focus on client development and compliance. Possibly narrow scope of expected use-cases (e.g. IM vs. generic pub-sub). - C2S and S2S improvements, possibly backwards-incompatible. Looking towards better mobile compatibility.

  111. theTedd

    > are the notes funny? if yes, it was Tedd wasn't me; I'm still catching up

  112. Ge0rG

    Ketchup!

  113. theTedd

    Mustard!