Forwarded message from Danny O'Brien on Wed Aug 11, 2021 at 3:15 PM:
Hiya!

Sorry, yes -- not so much in my spam folder, but in my terrifyingly full
inbox.

First up, here's an Estuary invite:
https://estuary.tech/sign-up -- if you post this
to the public list, could you remove this link first? Even if you beat them
to using it, my next invite might be... a little easy to guess :)

Questions answered in-text below:

On Thu, Aug 5, 2021 at 9:59 AM Nguyễn Gia Phong <mcsinyx@disroot.org> wrote:

> Dear Danny O'Brien,
>
> > I saw you talking about your python wheels repository on IPFS -- I have a
> > few invites for Estuary.tech, which is Protocol Labs' experiment in
> > providing filecoin-funded fallback pins for IPFS.  If you use it, it'll
> > save any of your CIDs to the filecoin network for free, and if they drop
> > offline, restore them from those backups and keep them up.
>
> Thank you for the invitation!  We would love to make best use of it,
> although it's worth noting that IPWHL (at its current stage) is also
> more or less an experiment: the entire collection has under 200 wheels
> at the moment and its concept is yet to be discussed with many people
> working on (Python) packaging and/or IPFS.  We have high hope for it,
> but I thought you would like to know the "risks" (-;
>

Oh absolutely -- part of why I thought you'd be good for this is that while
there's a lot of opportunity here, I'm not going to blow up PyPi if we run
into teething problems :)

Also I've been seeking out projects that are in the earliest stages, so we
can see where it helps (if it helps) when you're starting from scratch.
Obviously a lot of the Filecoin work right now is hooking into people's
existing backup/storage systems, but I don't want us to miss chances where
there's a more natural IFPS/Filecoin approach to doing things. As an
example, an IPFS-first project obviously worries less about hooking up into
a traditional CDN, but does that mean it *never* needs to think about a
CDN? Is it easier to hook in, say, a BitTorrent/WebTorrent backend if you
start with IFPS first? Etc, etc.



> > Happy to answer questions -- basically, my job is encourage and suppport
> > decentralized web services, so really pleased that you're working in this
> > area.
>
> One thing I am curios about Filecoin though, is that how well distributed
> is it at the moment?  To be frank IPFS content discoverability is rather
> high in latency for less popular content so we are connecting directly
> to the pinning node in order to avoid pip from timing out [0].
>
>
First up, I should say that I work for the Filecoin Foundation - as well as
the non-profit the Filecoin Foundation for the Decentralized Web, which
confusingly is another organization entirely, but which a lot of FF
employees donate time to... -- but *not* Protocol Labs. FF's mission is to
steward the Filecoin ecosystem as a whole, FFDW's mission is to support the
*rest* of the decentralized stack, especially those bits that can help with
the storage of the world's most important knowledge.

Anyway, both orgs are independent of Protocol Labs, who are building out
IPFS, Filecoin, and Estuary. However, I see part of my job is to poke PL
and find out what's going on, including finagling them to give me some
invites to give out to FLOSS projects to see what they think. What I
definitely don't have is deep official insight into PL plans, or any power
over those plans apart from my tentative abilities to sweet-talk them into
supporting worthy projects :)

Estuary is an Protocol Labs experiment (out of their application research
group, https://arg.protocol.ai/ ) in what it would look like to have a more
seamless link between Filecon-stored data and the public IPFS network
Basically how it works is that Estuary checks to see if it can see your
CIDs on the network, shares them if it can see them, and if not it'll pull
them from the (free) Filecoin backups -- so you'll always have at least one
pinned source, even if all your others die. See https://docs.estuary.tech/

What I *don't* know is how many  IPFS nodes Estuary shares that data on.
It's just shipped, so they're playing around with this a lot. I did spot
that it's now got some hook up with https://fission.codes/  which is
another, independent, IPFS hosting service with some cool features; so they
may be thinking of linking it more widely to the existing pinning clusters.


> It would be lovely if we don't have to rely on a public location address,
> but it took exceptionally long for a non local node to find the content
> on my personal machine, and even when Pinata pins shows up instantly
> on many public gateway, it can still take minutes for the CI's node to
> find it.  I am thinking towards collaborative clusters [1] in a more
> distant future; would Filecoin be a better alternative in term of
> content discoverability and is there a way to avoid all pinners
> being close geographically?
>
>
So not in Filecoin 1.0 as it were -- As I understand it, Protocol Labs is
still working on the "retrieval market
<https://spec.filecoin.io/systems/filecoin_markets/retrieval_market/>" bit
of Filecoin. It's a tricky problem. Basically -- the current Filecoin
network uses compact mathematical proofs to show that a Filecoin storage
provider is really storing your data (and how many copies they are
storing), but doing the same thing with proving that certain data is
assuredly online isn't so straightforward. Filecoin is definitely pretty
well-distributed geographically (though with the center of gravity
definitely in China), but they're not necessarily putting their stored
files on IPFS. It may be that with tools like Estuary they'd be more
tempted to do that, but they'd currently have to settle a deal with you to
do that outside of the Filecoin network.

OTOH I think this is as much an IPFS issue as a Filecoin one. I'm talking
to the IPFS folks next week to better understand some of these pain points.
I'll echo back your feedback; it's a pretty common one.



> Either way, we would be grateful for any leverage we could have
> right now.  In addition, would it be OK for me to forward and continue
> this discussion on IPWHL public mailing list [2]?
>
>
 Sure -- could you take out the invite link before you forward it?

Best wishes,
> Nguyễn Gia Phong
>
> [0] https://git.sr.ht/~cnx/ipwhl-data/tree/main/item/.build.yml
> [1] https://collab.ipfscluster.io
> [2] ~cnx/ipwhl-discuss@lists.sr.ht

On Wed Aug 11, 2021 at 3:15 PM -0700, Danny O'Brien wrote:
> Basically how it works is that Estuary checks to see if it can see your
> CIDs on the network, shares them if it can see them, and if not it'll pull
> them from the (free) Filecoin backups -- so you'll always have at least one
> pinned source, even if all your others die. See https://docs.estuary.tech/

Thanks, this is very interesting!

> So not in Filecoin 1.0 as it were -- As I understand it,
> Protocol Labs is still working on the "retrieval market" [0]
> bit of Filecoin.

How does this work when a client is from IPFS public network,
i.e. should I be worried about the following?

> The client receives the request for payment.

> Filecoin is definitely pretty well-distributed geographically
> (though with the center of gravity definitely in China), but they're
> not necessarily putting their stored files on IPFS.

I noticed that estuary.tech is hosted in North America.  It would be
unfortunate if the backing Filecoin nodes are not in the IPFS public DHT.

> It may be that with tools like Estuary they'd be more tempted to do that,
> but they'd currently have to settle a deal with you to do that outside
> of the Filecoin network.

Is there a technological reason for this, or it's simply not economical
to do so?  The interoperation briefly mentioned in the Filecoin docs [1]
is not clear enough for me to understand.

> OTOH I think this is as much an IPFS issue as a Filecoin one. I'm talking
> to the IPFS folks next week to better understand some of these pain points.
> I'll echo back your feedback; it's a pretty common one.

Thank you, I would be grateful to know what they have in mind to solve it.

> > In addition, would it be OK for me to forward and continue
> > this discussion on IPWHL public mailing list?
>
> Sure -- could you take out the invite link before you forward it?

Thanks, I've removed the token before forwarding.  Note that the list
insists on https://useplaintext.email and bounces back all HTML emails.

[0] https://spec.filecoin.io/systems/filecoin_markets/retrieval_market
[1] https://docs.filecoin.io/about-filecoin/ipfs-and-filecoin/#using-ipfs-and-filecoin

Dear Danny O'Brien,

> Hullo! I was just going through old emalis, and realised I left you hanging
> with a few questions. Hope it's all going well with you!

Thanks!  I'm doing great and wishing you the same.

> > > So not in Filecoin 1.0 as it were -- As I understand it,
> > > Protocol Labs is still working on the "retrieval market" [0]
> > > bit of Filecoin.
> >
> > How does this work when a client is from IPFS public network,
> > i.e. should I be worried about the following?
>
> Nope, Estuary acts as a caching IPFS node — essentially, it will reload its
> cache from the existing network for a CID, or if there are no more nodes
> out there, it will load the cache from the Filecoin network. There's no
> cost involved in this, and the storage is free because the cryptoeconomics
> of Filecoin right now mean that the cost of  useful, public data is heavily
> discounted. (I can explain this at more length, if you're interested.)
>
> > > Filecoin is definitely pretty well-distributed geographically
> > > (though with the center of gravity definitely in China), but they're
> > > not necessarily putting their stored files on IPFS.
> >
> > I noticed that estuary.tech is hosted in North America. It would be
> > unfortunate if the backing Filecoin nodes are not in the IPFS public DHT.
>
> The backing Filecoin nodes aren't in the IPFS DHT — they just store the
> data, they don't act as distributors of it. If you need to pin your CIDs
> more widely, then you could do that with other services (although it's the
> nature of IPFS that it shouldn't matter where your pins are relative to
> your users after the first retrieval — your users will act as nodes for the
> content for a while after that.)
>
> > > It may be that with tools like Estuary they'd be more tempted to do that,
> > > but they'd currently have to settle a deal with you to do that outside
> > > of the Filecoin network.
> >
> > Is there a technological reason for this, or it's simply not economical
> > to do so? The interoperation briefly mentioned in the Filecoin docs [1]
> > is not clear enough for me to understand.
>
> Technological atm — there's no way to reliable conduct a automatic proof of
> distribution in the way that you can prove that the file is being stored
> with the proof of spacetime.

Thank you for the explanations.  I am quite satisfied with them
and I am forwarding them to the mailing list [2] for later references.

Best wishes,
Nguyễn Gia Phong

[0] https://spec.filecoin.io/systems/filecoin_markets/retrieval_market
[1] https://docs.filecoin.io/about-filecoin/ipfs-and-filecoin/#using-ipfs-and-filecoin
[2] https://lists.sr.ht/~cnx/ipwhl-discuss/%3CCDHEYS0GBRZR.14KH5733I4K0J%40nix%3E