jdev - 2022-05-17


  1. qy

    Is there an XEP for file sharing that isn't http-based, that is actually well supported among clients, yet?

  2. Zash

    The answer is something like "of these three properties you want, pick any two"

    👍 1
  3. qy

    😔

  4. Zash

    There's Jingle FT, which is supported, not sure how widely exactly. It was plagued by interop problems before, tho might have settled by now. Doesn't work with multiple devices or MUC tho. Transfer methods have reliability issues.

  5. qy

    :/

  6. Kev

    Why would you not want it to be HTTP-based?

  7. Zash

    There are others like https://xmpp.org/extensions/xep-0385.html and https://xmpp.org/extensions/xep-0447.html but I don't think support is there yet.

  8. Zash

    Obligatory confusion between negotiation and transfer!

  9. qy

    I'm mostly just curious. I know theres a tonne of XEPs around, at least two of which i really want to go somewhere fast

  10. pep.

    Goffi has a jingle component fwiw

  11. pep.

    Most of the benefits of http upload isn't that it's http anyway, it's that it's a component

  12. pep.

    And it can do offline / multiparty stuff

  13. Zash

    Store and forward not dependent on origin availbility.

  14. Zash

    Component?

  15. pep.

    Also

  16. pulkomandy

    I am disappointed to not see http://samy.pl/pwnat/ used more often for p2p things (and in this case it could be a simplified version of it if the xmpp connection can be used to agree on tcp or udp port numbers)

  17. pulkomandy

    The basic idea is "send packets pretending that a connection is already open, and since the NAT/firewall on either side usually assumes all outgoing connections are allowed, it will actually work"

  18. Zash

    Write a XEP? But then, p2p and multi-party distribution is still an issue

  19. Zash

    Bittorrent is another thing. If you rope in the server as a peer you might get similar benefits as http upload

  20. pep.

    hot take: http upload is only good as a negociated transport method behind jingle anyway :-°

  21. pep.

    hot take: http upload is only good as a negociated transport behind jingle anyway :-°

  22. qy

    Im slightly keen to have ipfs as the backing proto for file transfer, but that'd need servers as ipfs nodes

  23. qy

    Or maybe not, clients could be

  24. qy

    Actually realistically it'd need neither, but that'd help for better resolution

  25. Zash

    p2p and mobile is also problematic wrt battery usage

  26. qy

    So yeah, serverside

  27. qy

    Possible to use go-ipfs externally as the server, and hook in by a mod_ipfs, and expose that to clients as some ipfs-native representation, and also http upload

  28. qy

    I know people like bittorrent, but i really want ipfs to work, and to do that it just needs more projects with it baked in

  29. qy

    But in this case that'd need an xep probably and i don't wanna get bogged down in beaurocracy if nobody else likes the idea

  30. msavoritias

    But whats the benefit since the server is gonna be involved either way 🤔

  31. flow

    it's often *not* people not liking the idea, but people do not have the resources to explore and implement it

  32. moparisthebest

    msavoritias, what do you mean? this is XMPP, the server is always involved

  33. flow

    it's often *not* people not liking the idea, but people not having the resources to explore and implement it

  34. msavoritias

    moparisthebest: i was talking about what qy said with ipfs

  35. moparisthebest

    right but how does it change anything I mean

  36. qy

    msavoritias: the server hosts the original and seeds to ipfs, but clients dont need to fetch it from the server, they fetch it from any ipfs gateway or ipfs native, and if the server goes down, it's all still there because ipfs caching

  37. moparisthebest

    well, it *may* still be there

  38. moparisthebest

    qy, but I like the concept, sounds like you can base much of the XEP off http upload, give it a shot ?

  39. moparisthebest

    it's not so much bureaucracy

  40. msavoritias

    Not to be a downer but this seems incompatible with gdpr. Unless specific case

  41. moparisthebest

    not sure why it would be, you upload your file knowing all this in advance

  42. moparisthebest

    if it is, then so is XMPP, because you sent that message not knowing how long I'll keep it (might be forever)

  43. msavoritias

    The difference is that in XMPP its not known how long true. With ipfs though its by design to be kept forever.

  44. msavoritias

    Or at least they try to

  45. moparisthebest

    not *really*, it's just as long as each node caches it, which in practice isn't so long

  46. moparisthebest

    at least last time I played with it a couple years ago...

  47. msavoritias

    That is what they advertise thougb

  48. moparisthebest

    try it, run an ipfs node with your website on it, when you turn your node off your website will be gone

  49. msavoritias

    > moparisthebest wrote: > not sure why it would be, you upload your file knowing all this in advance You expect it to be on the server and the recipient/s With ipfs its in the server, the recipients and an unknown number of computers which have nothing to do with your server or the people you talk to

  50. msavoritias

    moparisthebest: as stated. It could be. But they advertise it for resiliency.

  51. msavoritias

    Just like most xmpp operators are not honey pots

  52. moparisthebest

    as far as I know ipfs doesn't really have a pre-fetching thing by default, you can only fetch things you know the unique hash of

  53. qy

    exactly

  54. qy

    nothing is fetched until advertised by a seeking node

  55. qy

    so hence, nothing is cached unless wanted

  56. Zash

    What would that do for latency?

  57. qy

    and caching is stronger with things more wanted

  58. qy

    it would be shit latency for anything not heavily cached, hence shit latency for anything not heavily wanted

  59. qy

    but, the latency is almost always just a one-time thing, that's why for my ipfscat script, i prefetch from public gateways, so once it's on one it's pretty much on them all, and once it's there it's not going away for at least a day, usually longer

  60. qy

    with a bunch of xmpp nodes on it, i'd wager it'd get a lot better anyway

  61. qy

    the ipfs network provenly gets better with more nodes

  62. pulkomandy

    what would that be used for in xmpp world? sharing a picture with a MUC? that's maybe ~1000 users max? or do you have some more interesting applications in mind? or is this just offtopic discussion now? (not that I mind offtopic things, just trying to follow the ideas)

  63. qy

    i'm mainly thinking just any random file sharing. i don't like HTTP and i'd rather support IPFS over that

  64. Zash

    the access pattern with http upload from what I can tell is that most hits is just after the referencing message was sent, then maybe the odd hit a bit later. something that improves with time doesn't sound great for that

  65. qy

    i already kinda use ipfs for file sharing anyway

  66. qy

    Zash, that's the thing, if servers are nodes, it wouldn't matter

  67. qy

    if servers are nodes, they'll end up well connected to cdn gateways, because so much file sharing is occuring, so latency would be next to none in real terms

  68. qy

    it WOULD "improve with time", but in realistic terms, "time" would be pretty short because the links between servers and e.g. cloudflare-ipfs.com, would pretty much never be dropped due to constant access

  69. pulkomandy

    I don't really understand, for file sharing over XMPP usually I just want to send one file from one sender to one receiver, and I think I wouldn't want the files to be stored on random ipfs nodes along the way?

  70. qy

    it would be terrible if servers weren't nodes, yes

  71. qy

    pulkomandy, imagine bittorrent, but ephemeral. that's all this is

  72. pulkomandy

    yes but that's what ipfs is already, so I don't see the link with xmpp

  73. qy

    the link is just that more nodes makes ipfs better, and more ipfs makes xmpp more file-resilient

  74. qy

    e.g. if a server goes down, currently, that makes all files "Gone"

  75. qy

    ipfs: that's not the case

  76. pulkomandy

    I consider that being able to delete a file easily and being sure it is not still there on some server is a feature. I guess we have very different uses for file sharing

  77. qy

    deleting anything on the internet is always a misfeature

  78. pulkomandy

    so we should preserve everything and drown in a sea of outdated and irrelevant information?

  79. Zash

    Servers probably don't go down in the few second window when the wast majority of downloads occur.

  80. qy

    nothing is preserved indefinitely, but ipfs allows elastic caching that means one server going down is next-to irrelevant

  81. moparisthebest

    you can't delete anything you share on the internet, that's a rule of the internet

  82. moparisthebest

    despite GDPR whining

  83. qy

    ^

  84. moparisthebest

    https://en.wikipedia.org/wiki/Streisand_effect

  85. pulkomandy

    as long as someone is interested in it

  86. pulkomandy

    my quest to recover and archive lost BeOS software says as much. Most of it was not on the internet anymore. People trying to track down software for PalmOS would say as much too

  87. qy

    i'm keen to get ipfs more public and embedded in more software. as long as servers are interested in adding it, and the xep process wouldn't be too beaurocratic and people are happy for it to be a thing, i don't mind writing one, but i won't bother if it won't be used

  88. pulkomandy

    also, my private conversations in XMPP are not exactly "the internet"

  89. qy

    i'm happy to write an xep and a mod_ipfs for prosody in lua, at best

  90. qy

    maybe an erlang module at a stretch

  91. Zash

    I wonder if this couldn't be done with HTTP Upload and OOB?

  92. qy

    i mean, it could, with custom p2p, but then you'd be forsaking the existing ipfs p2p network

  93. qy

    it would be a no brainer to use the existing p2p network for extra dataresilience

  94. Zash

    As in, server provides a HTTP upload interface that gets stored into ipfs

  95. qy

    oh

  96. Zash

    Return ipfs:// URL

  97. qy

    probably. i've been eyeing that for some time, the http external upload interface is woefully incompatible, and http upload trivially so without modification

  98. Zash

    Or return some https://well-known-ipfs-gateway.example/ URL

  99. qy

    but i'd also like to be able to modify one of those to suit, if happy

  100. qy

    up to you, as the server dev

  101. qy

    if prosody supports it, i'm sure more could happen

  102. Zash

    Prosody supports it if you write a XEP-0114 external component 😉

  103. Zash

    or a module 🙂

  104. qy

    wait, wouldn't require an xep?

  105. qy

    i'm in

  106. qy

    would you be happy to advertise it as much as generic http upload?

  107. qy

    (i'm thinking just some comment in the default config, at best)

  108. Zash

    If it's done as a thing that exposes a http upload service but stores the file in ipfs, I'm not sure if any protocol changes are needed

  109. moparisthebest

    a module that accepts https uploads and returns https URLs to ipfs gateways would work without any protocol work easily

  110. moparisthebest

    but is that what you want ?

  111. qy

    fab, i'll try it

  112. qy

    at a start, it's better than nothing

  113. Zash

    PoC or more thinking over a whiteboard might help, but now I'm going to be busy with pancake coma

  114. qy

    i can then rework that to include something more embedded with ipfs later

  115. Zash

    If the client has ipfs, it could just put stuff there directly and send, no server needed. Servers could provide (via XEP-0215 maybe or some other discovery method) ipfs gateways locally.

  116. qy

    I'll see what i can draft

  117. moparisthebest

    yea that seems more like what you want, and 0 server work required

  118. moparisthebest

    send a regular "I uploaded this https file for you" @ an ipfs gateway for backward compat, with an added "or fetch with ipfs here" element, done

  119. Zash

    Here's where SIMS or similar would be nice, as it allows alternative transfer methods to be listed

  120. qy

    would it be more forward-thinking to implement it over SIMS?

  121. Zash

    SIMS is what the sending client would say, so if you're implementing both client and server bits then it might be sensible to look into

  122. Zash

    That is, https://xmpp.org/extensions/xep-0385.html or maybe https://xmpp.org/extensions/xep-0447.html