Decentralized substitute distribution with ERIS

Decentralized substitute distribution with ERIS v2 PROPOSED

Hello Guix,

Here comes the V2 of a proposal towards decentralizing substitute distribution
with ERIS.

A quick summary (as this has become quite long):

- This adds support for publishing and getting substitutes over IPFS.
- By using the ERIS encoding we are not limited to using IPFS as transport. We
can also use GNUNet, Named Data Networking (possibly) or just plain old HTTP. Support
for these can be added in (guix eris).
- These patches are still very rough and we need better logic for when to use
IPFS et. al. and when to fallback to HTTP.
- There might be performance issues when using IPFS via the IPFS daemon HTTP
API.

I found the setup for testing this a bit tricky. I will try and describe how I
have been testing it. Please let me know how this can be improved!

** Authorize local substitutes

We will be running a local substitute server so we need to add the local
signing key to the list of authorized keys. In the system configurations:

#+BEGIN_SRC scheme
(modify-services %base-services
(guix-service-type
config =>
(guix-configuration
(inherit config)
(authorized-keys
(cons*
;; allow substitutes from ourselves for testing purposes
(local-file "/etc/signing-key.pub")
%default-authorized-guix-keys)))))
#+END_SRC

** Configure the local Guix checkout

#+BEGIN_SRC shell
./bootstrap && ./configure --localstatedir=/var --sysconfdir=/etc
#+END_SRC

The ~--sysconfdir~ is required so that guix will use the ACL in ~/etc/guix/acl~.

** Start the IPFS daemon

#+BEGIN_SRC shell
guix shell go-ipfs -- ipfs daemon
#+END_SRC

Start a local substitute server:

#+BEGIN_SRC shell
sudo -E ./pre-inst-env guix publish --public-key=/etc/guix/signing-key.pub --private-key=/etc/guix/signing-key.sec --cache=/tmp/guix-publish-cache/ --port=8081 --compression=zstd:19
#+END_SRC

We use port 8081 as IPFS is running on 8080.

We use the temporary cache directory ~/tmp/guix-publish-cache~.

** Build some package locally

First we build some package:

#+BEGIN_SRC shell
./pre-inst-env guix build hello --no-substitutes --no-offload
#+END_SRC

#+RESULTS:
: /gnu/store/khaaib6s836bk5kbik239hlk6n6ianc4-hello-2.11

** Trigger the substitute server to "bake" a susbtitute

#+BEGIN_SRC shell
curl http://localhost:8081/khaaib6s836bk5kbik239hlk6n6ianc4.narinfo
#+END_SRC
--8<---------------cut here---------------start------------->8---
StorePath: /gnu/store/khaaib6s836bk5kbik239hlk6n6ianc4-hello-2.11
URL: nar/zstd/khaaib6s836bk5kbik239hlk6n6ianc4-hello-2.11
Compression: zstd
NarHash: sha256:11pk3jsh4zk0gigyjk881ay1nnvjfgpd3xpb4rmbaljhbiis4jbm
NarSize: 190480
References: 094bbaq6glba86h1d4cj16xhdi6fk2jl-gcc-10.3.0-lib 5h2w4qi9hk1qzzgi1w83220ydslinr4s-glibc-2.33 khaaib6s836bk5kbik239hlk6n6ianc4-hello-2.11
Deriver: mc7i1cdi42gy89mxl48nhdhgrfa9lpq6-hello-2.11.drv
Signature: 1;strawberry;KHNpZ25hdHVyZSAKIChkYXRhIAogIChmbGFncyByZmM2OTc5KQogIChoYXNoIHNoYTI1NiAjOTE0QTVGNTE4NUZGRUIzMzc4QTEwMzgzQzdFMEU1NDI1MEUyREZDRjk1RDUwOTNCMzU4QTFBNDE4OUFBRDVGNCMpCiAgKQogKHNpZy12YWwgCiAgKGVjZHNhIAogICAociAjMDkxMDA2NDlCMkMyMzhEQzE2ODhFQTgyQTdCOEJFMTc5MTVBMjVDQjc1NzcwQjlGRkNGOTFDRTg2MDgyNzAwQiMpCiAgIChzICMwMUFBQ0VERjY0N0VENTQyRTIwNENDMEM1M0VDMEY0QjQ4QzdEOTAyRkFEQTkxREI4NzRGQjE2MTQ4QTIzNUI2IykKICAgKQogICkKIChwdWJsaWMta2V5IAogIChlY2MgCiAgIChjdXJ2ZSBFZDI1NTE5KQogICAocSAjMDRDMkY4ODk1QTU0NDNGNTlCODk2NDEwMEI1MDY0NzU4RjQ1N0YzMENEREE1MTQyQzE0MDc0NjExNTA1NTc5MCMpCiAgICkKICApCiApCg==
--8<---------------cut here---------------end--------------->8---

If you do this again after a few seconds you will get a different response that
has the ERIS URN and the FileSizes. The reason for this is that Guix publish
bakes the nars asyncrhonisly in the background:

#+BEGIN_SRC shell
curl http://localhost:8081/khaaib6s836bk5kbik239hlk6n6ianc4.narinfo
#+END_SRC

#+RESULTS:
--8<---------------cut here---------------start------------->8---
StorePath: /gnu/store/khaaib6s836bk5kbik239hlk6n6ianc4-hello-2.11
URL: nar/zstd/khaaib6s836bk5kbik239hlk6n6ianc4-hello-2.11
Compression: zstd
FileSize: 57691
NarHash: sha256:11pk3jsh4zk0gigyjk881ay1nnvjfgpd3xpb4rmbaljhbiis4jbm
NarSize: 190480
ERISFormat: application/x-nix-archive+zstd-19
ERIS: urn:erisx2:B4AYPTXLTACB6WJYJ74RKBCVU3RBLHA4PY6HATUWRZNJ6THVSDUFM34K2ASUF3B6EOYEEBRZ5XEUR4PAAAIED7G7YSEZVZ5V7WWZ2PSC7Q
References: 094bbaq6glba86h1d4cj16xhdi6fk2jl-gcc-10.3.0-lib 5h2w4qi9hk1qzzgi1w83220ydslinr4s-glibc-2.33 khaaib6s836bk5kbik239hlk6n6ianc4-hello-2.11
Deriver: mc7i1cdi42gy89mxl48nhdhgrfa9lpq6-hello-2.11.drv
Signature: 1;strawberry;KHNpZ25hdHVyZSAKIChkYXRhIAogIChmbGFncyByZmM2OTc5KQogIChoYXNoIHNoYTI1NiAjOTE0QTVGNTE4NUZGRUIzMzc4QTEwMzgzQzdFMEU1NDI1MEUyREZDRjk1RDUwOTNCMzU4QTFBNDE4OUFBRDVGNCMpCiAgKQogKHNpZy12YWwgCiAgKGVjZHNhIAogICAociAjMDkxMDA2NDlCMkMyMzhEQzE2ODhFQTgyQTdCOEJFMTc5MTVBMjVDQjc1NzcwQjlGRkNGOTFDRTg2MDgyNzAwQiMpCiAgIChzICMwMUFBQ0VERjY0N0VENTQyRTIwNENDMEM1M0VDMEY0QjQ4QzdEOTAyRkFEQTkxREI4NzRGQjE2MTQ4QTIzNUI2IykKICAgKQogICkKIChwdWJsaWMta2V5IAogIChlY2MgCiAgIChjdXJ2ZSBFZDI1NTE5KQogICAocSAjMDRDMkY4ODk1QTU0NDNGNTlCODk2NDEwMEI1MDY0NzU4RjQ1N0YzMENEREE1MTQyQzE0MDc0NjExNTA1NTc5MCMpCiAgICkKICApCiApCg==
--8<---------------cut here---------------end--------------->8---

These patches have added the ERIS and ERISFormat fields. Eventually we would
have figured out what the best format for use over ERIS is, for now we encode
it in the ERISFormat field.

** Removing a package from the store

This is necessary in order to make guix look for a substitute.

#+BEGIN_SRC shell
./pre-inst-env guix gc -D /gnu/store/khaaib6s836bk5kbik239hlk6n6ianc4-hello-2.11
#+END_SRC

** Start the Guix daemon from the repository

#+BEGIN_SRC shell
sudo -E ./pre-inst-env guix-daemon --build-users-group=guixbuild --debug --substitute-urls=http://localhost:8081/
#+END_SRC

Note this will probably stop your system Guix daemon. Run ~sudo herd restart
guix-daemon~ to restart it.

#+BEGIN_SRC shell
./pre-inst-env guix build hello
#+END_SRC
--8<---------------cut here---------------start------------->8---
substituting /gnu/store/khaaib6s836bk5kbik239hlk6n6ianc4-hello-2.11...
downloading from urn:erisx2:B4AYPTXLTACB6WJYJ74RKBCVU3RBLHA4PY6HATUWRZNJ6THVSDUFM34K2ASUF3B6EOYEEBRZ5XEUR4PAAAIED7G7YSEZVZ5V7WWZ2PSC7Q ...
urn:erisx2:B4AYPTXLTACB6WJYJ74RKBCVU3RBLHA4PY6HATUWRZNJ6THVSDUFM34K2ASUF3B6EOYEEBRZ5XEUR4PAAAIED7G7YSEZVZ5V7WWZ2PSC7Q 502KiB/s 00:00 | 56KiB transferred

/gnu/store/khaaib6s836bk5kbik239hlk6n6ianc4-hello-2.11
--8<---------------cut here---------------end--------------->8---

We have just retreived the substitute for the hello package from IPFS. Hello
decentralized substitutes!

I have only tested this for fairly small packages (up to a few MB).

One issue with IPFS might be that we have to create a new HTTP connection to
the IPFS daemon for every single block (32KiB). The IPFS daemon does not seem
to support HTTP connection re-use and neither does the Guile (web client). I
fear this might become a performance issue. It seems possible to use IPFS more
directly by exposing the Go code as a C library and then using that with the
Guile FFI [1]. This is however a bit complicated and adds a lot of
dependencies. In particular, this should not become a dependency of Guix
itself. The performance of IPFS itself also needs to be evaluated, maybe the
IPFS HTTP API will not be the bottle-neck.

As mentioned in previous mail a simple HTTP transport for blocks would be a
good fallback. This would allow users to get missing blocks (things that
somehow got dropped from IPFS) directly from a substitute server. This is
different then getting the entire NAR from a substitute server. A user might be
missing a single 32KiB block and should be able to get only that. However, such
a HTTP fallback would also suffer from the one-connection-per-block issue. As
part of general ERIS research we are investigating CoAP as a better fallback
transport.

In any case, it would be necessary for the substitute server to store encoded
blocks of the NAR. For this I think it makes sense to use a small database. We
have bindings to use ERIS with GDBM [2]. It might also make sense to use
SQLite, especially if there are other use-cases for such a database.

I will be looking into the HTTP fallback and also using BitTorrent and GNUNet
as transports.

Thanks for making it so far and happy hacking!
-pukkamustard

[1] https://github.com/scala-network/libipfs/
[2] https://codeberg.org/eris/guile-eris/src/branch/main/eris/blocks/gdbm.scm

pukkamustard (5):
WIP: gnu: guile-eris: Update to unreleased git version.
publish: Add ERIS URN to narinfo
Add (guix eris).
publish: Add support for storing ERIS encoded blocks to IPFS.
substitute: Fetch substitutes using ERIS.

--
2.34.0

Maxime Devos <maximedevos@telenet.be>

~pukkamustard/eris

Decentralized substitute distribution with ERIS v2 PROPOSED

[RFC PATCH v2 1/5] WIP: gnu: guile-eris: Update to unreleased git version. Export this patch

[RFC PATCH v2 2/5] publish: Add ERIS URN to narinfo Export this patch

[RFC PATCH v2 3/5] Add (guix eris). Export this patch

[RFC PATCH v2 4/5] publish: Add support for storing ERIS encoded blocks to IPFS. Export this patch

[RFC PATCH v2 5/5] substitute: Fetch substitutes using ERIS. Export this patch