XSF Discussion - 2018-03-04

  1. SaltyBones

    wait icq is still useable?

  2. Zash

    Wait Telegram does everything right?

  3. Zash

    Should I even bother reading the rest?

  4. flow

    jonasw, after re-reading xep390 I wonder if the ASCII separators are valid in the separated strings itself. I know it likely qualifies as nit, but if they are valid then couldn't one could generate indistinguishable feature sets?

  5. daniel

    flow: they are forbidden in xml if that answers the question

  6. flow

    daniel, probably does, thanks

  7. flow

    although https://www.w3.org/TR/xml11/#NT-Char only discourages those

  8. daniel

    OK let me rephrase. It was the intention of the xep to use something that is forbidden in xml or xmpp (can remember which). I didn't actually check that myself. Maybe jonasw messed up 😀

  9. flow

    Na, it's probably a XML 1.0 vs 1.1 thingy

  10. flow

    in 1.1 the separators appear to be valid but discouraged, and in 1.0 they are invalid

  11. flow


  12. lovetox

    why do we have that in 0313

  13. lovetox

    These IDs are strings that servers may construct in any manner, and clients must treat as opaque strings (e.g. there is no requirement for them to be numeric, sequenced or GUIDs).

  14. lovetox

    is it not the most natural thing for a database to have a numeric index

  15. SaltyBones

    lovetox, might be since you have to query for timestamp and interface with different DBs it might look different

  16. lovetox

    SaltyBones, 1. you dont have to query with a timestamp

  17. lovetox

    2. this has nothing to do with why it would be some bad constraint for the server to have a numeric index as ID

  18. lovetox

    i bet every server has

  19. SaltyBones

    lovetox, http://logs.xmpp.org/xsf/2018-03-01/ 16:50

  20. SaltyBones

    lovetox, according to MattJ Prosody alone uses multiple formats.

  21. lovetox

    i know that servers use different formats, its just not clear why

  22. lovetox

    there is no faster search in an archive as with a numeric index

  23. lovetox

    also i see no need to encode other informations into the index if i have a row with columns that can hold these information

  24. lovetox

    as i see it using a numeric sequential index as id, would open up more possibilites with MAM, and i dont see the downside for the server

  25. SaltyBones

    Which possibilities

  26. lovetox

    it currently makes no sense to request a certain id from an archive

  27. lovetox

    because you dont know where to place it

  28. lovetox

    because there is no order information in the ID

  29. Ge0rG

    lovetox: it makes sense for ranged requests. Give me the first 50 messages after X. Then you'll have a new last ID of Y, and ask for the next 50 after Y

  30. lovetox

    thats how we use MAM now yes

  31. lovetox

    this would not change

  32. lovetox

    it doesnt depend on the ID beeing numeric or not, this works with any kind of ID

  33. lovetox

    and with numeric i mean sequenced

  34. Ge0rG

    lovetox: do you see any benefit in enforcing a certain implementation optimization into the XEP?

  35. jonasw

    flow, all XML parsers I tried rejected them even in entity form

  36. jonasw

    so it would at least be massively unsafe to use them in XMPP and get you disconnected very likely

  37. flow

    yep, because they are disallowed in XML 1.0 (but not 1.1)

  38. flow

    jonasw ^

  39. jonasw

    flow, yeah...

  40. jonasw

    RFC 6120 depends on XML 1.0 though

  41. flow

    I wonder if xep30 should specify string preperation for the various values

  42. flow

    jonasw, I'd love to add some rationale to xep390 explaining the situation. What do you think?

  43. jonasw

    flow, like this? https://xmpp.org/extensions/xep-0390.html#security-separators

  44. jonasw

    one could add a mention that RFC 6120 bases on XML 1.0 and not 1.1

  45. marc

    jonasw, what about merging my PRs for 401?

  46. jonasw

    can do that later today I guess

  47. jonasw

    remind me tonight maybe

  48. marc


  49. marc

    At least Ge0rG and Kev have no objections anymore :)

  50. moparisthebest

    Haha looking for backup solution and see jonasw commenting, do you still use Borg? :)

  51. jonasw

    moparisthebest, yes

  52. moparisthebest

    Btrfs send/receive solves all my problems except on this one machine that is ext4 :( Borg looks like a good candidate though

  53. flow is a happy borg user too

  54. Ge0rG

    moparisthebest: btrfs will solve all your problems by corrupting your data and laughing in your face.

  55. jonasw


  56. moparisthebest

    Ge0rG: my years of using it everywhere say otherwise, so far I've had corrupt (restore from backup) ext4 twice and only partially broken (still readable) btrfs once

  57. moparisthebest

    All those failures happened on a machine with a bad PSU that caused them btw

  58. Ge0rG

    moparisthebest: btrfs has some nice checksumming features, but they tend to fail on EMEM and have a huge scary warning instead of just fixing their RAID: https://btrfs.wiki.kernel.org/index.php/RAID56

  59. Ge0rG

    jonasw: thanks for #585

  60. marc

    jonasw, thanks for merging

  61. moparisthebest

    Oh yea 5/6 has never been useable, 0/1/10 has been solid for years though

  62. Ge0rG

    moparisthebest: not if you want to have 16TB of data on a 2GB RAM machine.

  63. moparisthebest

    I have a 32tb array on a 16gb machine and it uses hardly any ram

  64. moparisthebest

    I can't imagine running a server with 2gb ram

  65. Zash

    All my servers have 2GB RAM

  66. SamWhited

    Most of my servers have 1 GB of ram…

  67. moparisthebest

    Giant storage servers?

  68. SamWhited

    No, I don't remember how much is in my big storage server. It's probably 2 or 4.

  69. moparisthebest

    I mean if you are saying btrfs uses more memory then ext4 then sure, Linux uses more memory than an Arduino too, it's got a few more features though

  70. SamWhited

    I also have never had a problem with memory due to ZFS (which is broadly similar and people use the same complaint about it), so I dispute both of these arguments (that 16 gigs of ram can be expected, but also that the filesystem uses up too much ram)

  71. moparisthebest

    Yea I agree I don't think btrfs uses much ram

  72. Ge0rG

    SamWhited: the usual ZFS tutorials all begin with a huge red warning that your data is already as good as gone if you don't have 16GB or more RAM

  73. SamWhited

    Depends how big your storage pool is; 1GB of RAM per TB of storage is a good general guess for how much space you'll want for a cache.

  74. Ge0rG

    SamWhited: and regarding bitter btrfs experiences, I happen to have a documented case: https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg48375.html

  75. Ge0rG

    ...with too little RAM

  76. Ge0rG

    SamWhited: the question is actually how much *kernel memory* is needed to operate the fs, because you can't swap the kernel

  77. moparisthebest

    I don't run swap either meh

  78. SamWhited

    oh yah, that's poor

  79. marc

    Does Cisco Jabber comply with the XMPP standard?

  80. Maranda


  81. Ge0rG

    marc: at least on cisco.com they still dont have TLS

  82. Zash

    Looks like they do on c2s?

  83. marc

    Ge0rG, is Cisco Jabber SaaS or do they provide a server software?

  84. stuxnet

    Security and Cisco - what do you dream at night?

  85. Maranda

    marc, [19:58:22] ‎Echo1‎: Maranda: cisco.com is running jabberd version on an unknown platform

  86. marc


  87. marc


  88. Maranda

    marc, don't ask me what kind of a hack of jabberd14 that is because I don't know and I don't want to know.

  89. Maranda

    jabberd14 or jabberd that is.

  90. marc

    Maranda, according to Holger it is not related to the jabberd project all atl

  91. Zash

    Jabber XCP or something

  92. Maranda

    Zash, who knows. https://www.cisco.com/c/dam/en/us/products/collateral/unified-communications/jabber-windows/jabber-xcp-retirement-message.pdf

  93. Holger

    Yeah it's no longer called XCP.

  94. Holger


  95. Holger

    marc: They sell the server software.

  96. Holger

    (Usually integrated with their VoIP stuff these days.)

  97. marc

    Holger, ah okay. yes, the VoIP stuff is the reason why some people want to move from ejabberd(!) to cisco jabber ôÔ

  98. Holger


  99. Maranda facepalms.

  100. marc

    Because it works well with the Cisco Phones :D

  101. marc


  102. Holger

    Yes I get how this is nice for the enterprise use case.

  103. Holger

    Unfortunately their XMPP support degraded over the past few years.

  104. Holger

    They never implemented the modern XEPs and now slowly come up with proprietary extensions to support the multi-device/mobile stuff.

  105. marc

    Holger, does it mean that the phones will use VoIP via the jabber server or what's the use case?

  106. Holger

    marc: No it's mostly just an integrated address book for both (which can be Microsoft AD) and that you see the phone presence (or initiate calls) in your chat app. Stuff like that.

  107. Holger

    Their VoIP is just SIP.

  108. marc

    Holger, Sounds like very boring stuff to me. What does phone presence mean?

  109. Holger

    You're currently in a call or not.

  110. Holger

    My co-workers love it.

  111. Holger

    Well the administration people.

  112. Holger

    Staff council not so much.

  113. Zash

    Hmm, something XMPP-optimized similar to https://tools.ietf.org/html/draft-ietf-acme-email-tls-02 ?

  114. Kev

    I feel I'm missing something with this MAM thread, as he seems to be asking for a whole bunch of things that MAM can already do.

  115. Kev

    Has anyone else got a better grasp on what he's asking? He asked me in private too, and I suggested he take it to the list, but it's not explained it to me any better.

  116. Zash

    If you have some random MAM id, how do you retrieve that specific message?

  117. Kev

    If you have some random MAM id, why do you need that particular message? The holes he talks about are bounded by ids, and you can request the messages between two ids.

  118. lovetox

    he just finds it inefficient

  119. Kev

    Yes, that's the confusing thing. If you want the messages, then first asking for the ids and then asking for the messages is obviously inefficient.

  120. Kev

    Assuming you're getting the same ids and messages in both cases - which you seem to be able to do for all his use cases.

  121. lovetox

    i guess he doenst want to load all these messages

  122. Kev

    He says he does.

  123. lovetox

    instead he wants to load them later when looked at or something like that

  124. lovetox

    also i think he has plotted out his client architecture without looking how the xep works first

  125. waqas

    I skimmed through the thread. His mental model is a bit different than our normal one.

  126. waqas

    His model makes sense if this was REST/HATEOS

  127. waqas

    Or in cases where IDs have some significance or order is missing

  128. waqas

    But MAM isn't syncing of a key-value store, its syncing of a log, where content matters and IDs only play a supporting role.

  129. waqas

    MAM is like git, and unlike e.g., file system sync.

  130. Zash

    How is MAM like git?

  131. Kev

    It's sequenced.

  132. Kev

    You use the content of one item to find the previous item, linked-list stylee.

  133. waqas

    Yes, and you'd notice similarities if you look at the git/hg/etc sync protocols

  134. waqas

    Where commit hashes (ids) are used more like database cursors, even if you'd otherwise think of them as database primary keys.

  135. waqas

    This is very common in log sync protocols

  136. Zash

    I'd like to think that MAM is much simpler than dvcs sync protocols

  137. Kev

    I'm not sure why. Optimisations aside, git sync is very straightforward.

  138. Kev

    (And then the optimisations and details make it all much less pleasant, naturally)

  139. waqas

    Yes, git sync is very simple at its core

  140. Zash

    I don't know how git sync works. I only read the mailing list posts where it was said to be just scp/rsync/whatever

  141. waqas

    rsync is different, and more like what that person wants

  142. Zash

    By "rsync" I mean "just transfer the files"

  143. Zash

    If it were truly a content-addressable thing, that would work

  144. waqas

    The protocol requires both sides to scan the entire dataset in rsync :)

  145. waqas

    Because there is no order

  146. Zash

    But that'd be rsyncs problem, not git

  147. waqas

    Yep, because git can have a much simpler protocol due to commit ordering, "give me everything since this hash"

  148. Kev

    a la MAM

  149. Zash

    I'd imagine it'd be "between this hash and this branch/label/named commit"

  150. waqas

    Other examples of log sync protocols include kafka's sync, db replication protocols used by mysql/postgres/mongodb/etc all of which implement a write ahead log, etc

  151. waqas

    At their core if you ignore implementation specific bells and whistles, it's the same logic

  152. Zash

    Fetching stuff from one DAG to another ought to be more complicated

  153. Zash

    What I've seen of the hg protocol seems to agree

  154. waqas

    The main compleixity beyond MAM is multiple local and remote heads (or hg branches). It's largely the client communicating what it has and what it wants up to, give_me(from_hash_list, to_hash_list), plus some protocol elements and local git config to figure out what the server has.