~pukkamustard/eris

Decentralized substitute distribution with ERIS v2 PROPOSED

Hello Guix,

Here comes the V2 of a proposal towards decentralizing substitute distribution
with ERIS.

A quick summary (as this has become quite long):

- This adds support for publishing and getting substitutes over IPFS.
- By using the ERIS encoding we are not limited to using IPFS as transport. We
  can also use GNUNet, Named Data Networking (possibly) or just plain old HTTP. Support
  for these can be added in (guix eris).
- These patches are still very rough and we need better logic for when to use
  IPFS et. al. and when to fallback to HTTP.
- There might be performance issues when using IPFS via the IPFS daemon HTTP
  API.

I found the setup for testing this a bit tricky. I will try and describe how I
have been testing it. Please let me know how this can be improved!

** Authorize local substitutes

We will be running a local substitute server so we need to add the local
signing key to the list of authorized keys. In the system configurations:

#+BEGIN_SRC scheme
  (modify-services %base-services
    (guix-service-type
     config =>
     (guix-configuration
      (inherit config)
      (authorized-keys
       (cons*
        ;; allow substitutes from ourselves for testing purposes
        (local-file "/etc/signing-key.pub")
        %default-authorized-guix-keys)))))
#+END_SRC

** Configure the local Guix checkout

#+BEGIN_SRC shell
./bootstrap && ./configure --localstatedir=/var --sysconfdir=/etc
#+END_SRC

The ~--sysconfdir~ is required so that guix will use the ACL in ~/etc/guix/acl~.

** Start the IPFS daemon

#+BEGIN_SRC shell
guix shell go-ipfs -- ipfs daemon
#+END_SRC

Start a local substitute server:

#+BEGIN_SRC shell
  sudo -E ./pre-inst-env guix publish --public-key=/etc/guix/signing-key.pub --private-key=/etc/guix/signing-key.sec --cache=/tmp/guix-publish-cache/ --port=8081 --compression=zstd:19
#+END_SRC

We use port 8081 as IPFS is running on 8080.

We use the temporary cache directory ~/tmp/guix-publish-cache~.

** Build some package locally

First we build some package:

#+BEGIN_SRC shell
./pre-inst-env guix build hello --no-substitutes --no-offload
#+END_SRC

#+RESULTS:
: /gnu/store/khaaib6s836bk5kbik239hlk6n6ianc4-hello-2.11

** Trigger the substitute server to "bake" a susbtitute

#+BEGIN_SRC shell
curl http://localhost:8081/khaaib6s836bk5kbik239hlk6n6ianc4.narinfo
#+END_SRC
--8<---------------cut here---------------start------------->8---
StorePath: /gnu/store/khaaib6s836bk5kbik239hlk6n6ianc4-hello-2.11
URL: nar/zstd/khaaib6s836bk5kbik239hlk6n6ianc4-hello-2.11
Compression: zstd
NarHash: sha256:11pk3jsh4zk0gigyjk881ay1nnvjfgpd3xpb4rmbaljhbiis4jbm
NarSize: 190480
References: 094bbaq6glba86h1d4cj16xhdi6fk2jl-gcc-10.3.0-lib 5h2w4qi9hk1qzzgi1w83220ydslinr4s-glibc-2.33 khaaib6s836bk5kbik239hlk6n6ianc4-hello-2.11
Deriver: mc7i1cdi42gy89mxl48nhdhgrfa9lpq6-hello-2.11.drv
Signature: 1;strawberry;KHNpZ25hdHVyZSAKIChkYXRhIAogIChmbGFncyByZmM2OTc5KQogIChoYXNoIHNoYTI1NiAjOTE0QTVGNTE4NUZGRUIzMzc4QTEwMzgzQzdFMEU1NDI1MEUyREZDRjk1RDUwOTNCMzU4QTFBNDE4OUFBRDVGNCMpCiAgKQogKHNpZy12YWwgCiAgKGVjZHNhIAogICAociAjMDkxMDA2NDlCMkMyMzhEQzE2ODhFQTgyQTdCOEJFMTc5MTVBMjVDQjc1NzcwQjlGRkNGOTFDRTg2MDgyNzAwQiMpCiAgIChzICMwMUFBQ0VERjY0N0VENTQyRTIwNENDMEM1M0VDMEY0QjQ4QzdEOTAyRkFEQTkxREI4NzRGQjE2MTQ4QTIzNUI2IykKICAgKQogICkKIChwdWJsaWMta2V5IAogIChlY2MgCiAgIChjdXJ2ZSBFZDI1NTE5KQogICAocSAjMDRDMkY4ODk1QTU0NDNGNTlCODk2NDEwMEI1MDY0NzU4RjQ1N0YzMENEREE1MTQyQzE0MDc0NjExNTA1NTc5MCMpCiAgICkKICApCiApCg==
--8<---------------cut here---------------end--------------->8---

If you do this again after a few seconds you will get a different response that
has the ERIS URN and the FileSizes. The reason for this is that Guix publish
bakes the nars asyncrhonisly in the background:

#+BEGIN_SRC shell
curl http://localhost:8081/khaaib6s836bk5kbik239hlk6n6ianc4.narinfo
#+END_SRC

#+RESULTS:
--8<---------------cut here---------------start------------->8---
StorePath: /gnu/store/khaaib6s836bk5kbik239hlk6n6ianc4-hello-2.11
URL: nar/zstd/khaaib6s836bk5kbik239hlk6n6ianc4-hello-2.11
Compression: zstd
FileSize: 57691
NarHash: sha256:11pk3jsh4zk0gigyjk881ay1nnvjfgpd3xpb4rmbaljhbiis4jbm
NarSize: 190480
ERISFormat: application/x-nix-archive+zstd-19
ERIS: urn:erisx2:B4AYPTXLTACB6WJYJ74RKBCVU3RBLHA4PY6HATUWRZNJ6THVSDUFM34K2ASUF3B6EOYEEBRZ5XEUR4PAAAIED7G7YSEZVZ5V7WWZ2PSC7Q
References: 094bbaq6glba86h1d4cj16xhdi6fk2jl-gcc-10.3.0-lib 5h2w4qi9hk1qzzgi1w83220ydslinr4s-glibc-2.33 khaaib6s836bk5kbik239hlk6n6ianc4-hello-2.11
Deriver: mc7i1cdi42gy89mxl48nhdhgrfa9lpq6-hello-2.11.drv
Signature: 1;strawberry;KHNpZ25hdHVyZSAKIChkYXRhIAogIChmbGFncyByZmM2OTc5KQogIChoYXNoIHNoYTI1NiAjOTE0QTVGNTE4NUZGRUIzMzc4QTEwMzgzQzdFMEU1NDI1MEUyREZDRjk1RDUwOTNCMzU4QTFBNDE4OUFBRDVGNCMpCiAgKQogKHNpZy12YWwgCiAgKGVjZHNhIAogICAociAjMDkxMDA2NDlCMkMyMzhEQzE2ODhFQTgyQTdCOEJFMTc5MTVBMjVDQjc1NzcwQjlGRkNGOTFDRTg2MDgyNzAwQiMpCiAgIChzICMwMUFBQ0VERjY0N0VENTQyRTIwNENDMEM1M0VDMEY0QjQ4QzdEOTAyRkFEQTkxREI4NzRGQjE2MTQ4QTIzNUI2IykKICAgKQogICkKIChwdWJsaWMta2V5IAogIChlY2MgCiAgIChjdXJ2ZSBFZDI1NTE5KQogICAocSAjMDRDMkY4ODk1QTU0NDNGNTlCODk2NDEwMEI1MDY0NzU4RjQ1N0YzMENEREE1MTQyQzE0MDc0NjExNTA1NTc5MCMpCiAgICkKICApCiApCg==
--8<---------------cut here---------------end--------------->8---

These patches have added the ERIS and ERISFormat fields. Eventually we would
have figured out what the best format for use over ERIS is, for now we encode
it in the ERISFormat field.

** Removing a package from the store

This is necessary in order to make guix look for a substitute.

#+BEGIN_SRC shell
./pre-inst-env guix gc -D /gnu/store/khaaib6s836bk5kbik239hlk6n6ianc4-hello-2.11
#+END_SRC

** Start the Guix daemon from the repository

#+BEGIN_SRC shell
sudo -E ./pre-inst-env guix-daemon --build-users-group=guixbuild  --debug  --substitute-urls=http://localhost:8081/
#+END_SRC

Note this will probably stop your system Guix daemon. Run ~sudo herd restart
guix-daemon~ to restart it.

#+BEGIN_SRC shell
./pre-inst-env guix build hello
#+END_SRC
--8<---------------cut here---------------start------------->8---
substituting /gnu/store/khaaib6s836bk5kbik239hlk6n6ianc4-hello-2.11...
downloading from urn:erisx2:B4AYPTXLTACB6WJYJ74RKBCVU3RBLHA4PY6HATUWRZNJ6THVSDUFM34K2ASUF3B6EOYEEBRZ5XEUR4PAAAIED7G7YSEZVZ5V7WWZ2PSC7Q ...
 urn:erisx2:B4AYPTXLTACB6WJYJ74RKBCVU3RBLHA4PY6HATUWRZNJ6THVSDUFM34K2ASUF3B6EOYEEBRZ5XEUR4PAAAIED7G7YSEZVZ5V7WWZ2PSC7Q                                        502KiB/s 00:00 | 56KiB transferred

/gnu/store/khaaib6s836bk5kbik239hlk6n6ianc4-hello-2.11
--8<---------------cut here---------------end--------------->8---

We have just retreived the substitute for the hello package from IPFS. Hello
decentralized substitutes!

I have only tested this for fairly small packages (up to a few MB).

One issue with IPFS might be that we have to create a new HTTP connection to
the IPFS daemon for every single block (32KiB).  The IPFS daemon does not seem
to support HTTP connection re-use and neither does the Guile (web client).  I
fear this might become a performance issue. It seems possible to use IPFS more
directly by exposing the Go code as a C library and then using that with the
Guile FFI [1]. This is however a bit complicated and adds a lot of
dependencies. In particular, this should not become a dependency of Guix
itself. The performance of IPFS itself also needs to be evaluated, maybe the
IPFS HTTP API will not be the bottle-neck.

As mentioned in previous mail a simple HTTP transport for blocks would be a
good fallback. This would allow users to get missing blocks (things that
somehow got dropped from IPFS) directly from a substitute server. This is
different then getting the entire NAR from a substitute server. A user might be
missing a single 32KiB block and should be able to get only that. However, such
a HTTP fallback would also suffer from the one-connection-per-block issue. As
part of general ERIS research we are investigating CoAP as a better fallback
transport.

In any case, it would be necessary for the substitute server to store encoded
blocks of the NAR. For this I think it makes sense to use a small database. We
have bindings to use ERIS with GDBM [2]. It might also make sense to use
SQLite, especially if there are other use-cases for such a database.

I will be looking into the HTTP fallback and also using BitTorrent and GNUNet
as transports.

Thanks for making it so far and happy hacking!
-pukkamustard


[1] https://github.com/scala-network/libipfs/
[2] https://codeberg.org/eris/guile-eris/src/branch/main/eris/blocks/gdbm.scm

pukkamustard (5):
  WIP: gnu: guile-eris: Update to unreleased git version.
  publish: Add ERIS URN to narinfo
  Add (guix eris).
  publish: Add support for storing ERIS encoded blocks to IPFS.
  substitute: Fetch substitutes using ERIS.

 Makefile.am                         |  1 +
 configure.ac                        |  5 +++
 gnu/packages/guile-xyz.scm          | 10 ++---
 gnu/packages/package-management.scm |  1 +
 guix/eris.scm                       | 60 +++++++++++++++++++++++++++++
 guix/narinfo.scm                    | 14 +++++--
 guix/scripts/publish.scm            | 32 +++++++++++++--
 guix/scripts/substitute.scm         | 21 +++++++---
 8 files changed, 126 insertions(+), 18 deletions(-)
 create mode 100644 guix/eris.scm

-- 
2.34.0
pukkamustard schreef op di 25-01-2022 om 19:21 [+0000]:
Next
pukkamustard schreef op di 25-01-2022 om 19:21 [+0000]:
Next
pukkamustard schreef op di 25-01-2022 om 19:21 [+0000]:
Next
This (ipfs-daemon-alive?) seems racy, although it's probably not.
Can we do

(define guix-eris-block-reducer
  (case-lambda
    (() (guard (c (oops-it-fails-because-the-daemon-cannot-be-
contacted? c)
                  #false)
          (eris-block-ipfs-reducer))
    [...]))

instead? (I don't think this will work as-is, because from the name and
thunkiness, it would appear that eris-block-ipfs-reducer returns a
procedure ...

Greetings,
Maxime.
Next
pukkamustard schreef op di 25-01-2022 om 19:21 [+0000]:
Next
pukkamustard schreef op di 25-01-2022 om 19:22 [+0000]:
Next
pukkamustard schreef op di 25-01-2022 om 19:22 [+0000]:
Next
pukkamustard schreef op di 25-01-2022 om 19:22 [+0000]:
Next
pukkamustard schreef op di 25-01-2022 om 19:22 [+0000]:
Next
pukkamustard schreef op di 25-01-2022 om 19:22 [+0000]:
Next
Hi Maxime,

Maxime Devos <maximedevos@telenet.be> writes:
Next
Maxime Devos <maximedevos@telenet.be> writes:
Very interesting! I have been following your work on that a bit.

From what I understand gnunet-scheme interacts with the GNUNet services
and sends messages to the various GNUNet services. Is that correct?

Have you considered implementing the GNUNet protocols themeselves in
Guile? I.e. instead of connecting with the GNUNet services and sending
messages, implement R5N completely in Guile. IMHO this would be very
nice as one could use GNUNet protocols completely in Guile and not rely
on the GNUNet C code.

I believe this is somewhat the direction being taken with the GNUNet Go
implementation (https://github.com/bfix/gnunet-go) and also in line with
recent efforts to specify the individual GNUNet components and protocols
more independantly of one another (e.g. R5N is specified to work over IP
- https://lsd.gnunet.org/lsd0004/).

-pukkamustard
Maxime Devos <maximedevos@telenet.be> writes:
Next
Maxime Devos <maximedevos@telenet.be> writes:
Next
Maxime Devos <maximedevos@telenet.be> writes:
No, eris-encode will still return the ERIS URN. The blocks will just not
be stored in IPFS if the IPFS daemon is not running. See also my
response to the questions on guix-eris-block-reducer in (guix eris).

-pukkamustard
Maxime Devos <maximedevos@telenet.be> writes:
Yes, currently it is an unnecessary level of abstraction.

The idea is that when there are multiple backends/transports they are
multiplexed here. E.g. guix-eris-block-ref would attempt to use
HTTP/IPFS or whatever to get the block. Whatever calls
guix-eris-block-ref does not need to know from where the blocks come.

I hope to make this more clear in a V3 that will add HTTP transport.

-pukkamustard
Maxime Devos schreef op za 29-01-2022 om 22:38 [+0100]:
Next
Maxime Devos <maximedevos@telenet.be> writes:
Those are all possible reasons. I think we first need to do some
experiments to see if any of those claims can be justified.

In general, the logic for when to use ERIS transports needs more
thought.

For one, I think it should be user configurable. I can imagine that
certain users do not want to use decentralized substitutes at all. Users
should be able to deactivate the entire ERIS thing.

For the default, I personally think it would be ok to try and use
ERIS. The only thing we absolutely need is a clean fallback logic that
transparently falls back to getting the entire NAR by HTTP. Currently
such a fallback is not yet implemented.

-pukkamustard
Maxime Devos <maximedevos@telenet.be> writes:
No this should not work.

The ERIS URN is only used if the entire narinfo is signed with a
authorized signature. The FileHash is not used when getting substitutes
via ERIS (being able to decode ERIS content implies integrity).
The interesting case that would be allowed with ERIS is following:

1. Server A is authentic and its key is authorized.
2. Servers M1 to MN are potentially malicious and their keys are not
   authorized.
3. Server A and servers M1 to MN are in the substitute-urls.
4. Client gets Narinfo from server A and uses the ERIS URN from there.
5. Client can get blocks simultaneously from Server A and servers M1 to
   MN.
6. Client decodes content with the ERIS URN and can be sure that they
   have the valid substitute.

So client only needs to trust A but can use M1-MN (simultaneously) for
fetching the content.

-pukkamustard
Maxime Devos <maximedevos@telenet.be> writes:
Next
Hm, from what I understand connection re-use is something introduced in
HTTP/2 and go-ipfs does not do HTTP/2
(https://github.com/ipfs/go-ipfs/issues/5974).
Next
pukkamustard schreef op wo 02-02-2022 om 09:56 [+0000]:
Next
Yes, it works like the C GNUnet client libraries, except it's in Guile
Scheme and a few different design decisions were made, e.g. w.r.t.
concurrency.
Next
pukkamustard schreef op wo 02-02-2022 om 10:28 [+0000]:
The exception could be caused by, say:

  * an unbound variable
  * wrong arity
  * type error
  * stack overflow
  * prompt tag does not exist in current environment
  * out of memory

Except for the last one, these causes are all bugs and hence shouldn't
be surpressed.  Granted, this is a bit unlikely since this use of
'http-post' is very simple, but it's far from impossible for
(web client) to have a bug.

Greetings,
Maxime.
pukkamustard schreef op wo 02-02-2022 om 10:51 [+0000]:
Next
pukkamustard schreef op wo 02-02-2022 om 10:51 [+0000]:
Do we need a database at all?

E.g., if the client cannot download the data in the range [start, end]
because the corresponding block has disappeared, can it not simply
download that range from https://ci.guix.gnu.org/nar/[...]
(not sure about the URI) using a HTTP range request?

(Afterwards, the client should insert the block(s) back into
IPFS/GNUnet/whatever, maybe using this proposed ‘in-file block store’
such that other clients (using the same DHT mechanism) can benefit.)

Greetings,
Maxime.
pukkamustard schreef op wo 02-02-2022 om 11:10 [+0000]:
Next
Maxime Devos <maximedevos@telenet.be> writes:
Next
pukkamustard schreef op wo 02-02-2022 om 12:42 [+0000]:
Next
The client not only knows the ERIS URN, it also knows the location of
the nar (over classical HTTP) because it's in the narinfo.
Next
pukkamustard schreef op wo 02-02-2022 om 12:42 [+0000]:
If re-inserting missing blocks back into the IPFS/GNUnet/whatever is
made optional and is off by default, then almost nobody will enable the
‘caching peer’ option and we will have freeloaders, somewhat defeating
the point of GNUnet/whatever.

In a classic setting (‘plain old HTTP’), serving and downloading is a
separate thing.  But in a P2P setting, downloading cannot be separated
from uploading -- strictly speaking, a peer might be able to download
without uploading (depending on the P2P system), but that's anti-
social, not something that should be done by default.

However, if re-inserting missing blocks is _on_ by default, then there
doesn't seem to be any trouble.

Greetings,
Maxime.
Maxime Devos <maximedevos@telenet.be> writes:
Next
Export patchset (mbox)
How do I use this?

Copy & paste the following snippet into your terminal to import this patchset into git:

curl -s https://lists.sr.ht/~pukkamustard/eris/patches/28725/mbox | git am -3
Learn more about email & git

[RFC PATCH v2 1/5] WIP: gnu: guile-eris: Update to unreleased git version. Export this patch

* gnu/packages/guile-xyz.schm (guile-eris): Update to unreleased git version.
[source]: Update source URI.
[propagated-inputs]: Add guile-json-4 and guile-gdbm-ffi.
---
 gnu/packages/guile-xyz.scm | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/gnu/packages/guile-xyz.scm b/gnu/packages/guile-xyz.scm
index e2cf793acc..66ac486a74 100644
--- a/gnu/packages/guile-xyz.scm
+++ b/gnu/packages/guile-xyz.scm
@@ -4374,15 +4374,15 @@ (define-public guile-sodium
(define-public guile-eris
  (package
    (name "guile-eris")
    (version "0.2.0")
    (version "bcbbcbc88f3ec1f2fafcd034ce5b620516bff105")
    (source
     (origin
       (method git-fetch)
       (uri (git-reference
             (url "https://inqlab.net/git/eris.git")
             (commit (string-append "v" version))))
             (url "https://codeberg.org/eris/guile-eris")
             (commit version)))
       (file-name (git-file-name name version))
       (sha256 (base32 "1ijglmwkdy1l87gj429qfjis0v8b1zlxhbyfhx5za8664h68nqka"))))
       (sha256 (base32 "17v3h2hqx080739rl57nfradp5vlmy24fgqdxry1zal5z9d3i8sr"))))
    (build-system gnu-build-system)
    (arguments '())
    (native-inputs
@@ -4394,7 +4394,7 @@ (define-public guile-eris
           guile-srfi-180))
    (inputs (list guile-3.0))
    (propagated-inputs
     (list guile-sodium))
     (list guile-sodium guile-json-4 guile-gdbm-ffi))
    (synopsis "Guile implementation of the Encoding for Robust Immutable Storage (ERIS)")
    (description
     "Guile-ERIS is the reference implementation of the Encoding for Robust
-- 
2.34.0

[RFC PATCH v2 2/5] publish: Add ERIS URN to narinfo Export this patch

* guix/scripts/publish.scm: (bake-narinfo+nar): Compute ERIS URN of compressed nars.
(narinfo-string): Add #:eris-urn parameter and honor it.
* guix/scripts/narinfo.scm: (<narinfo>)[eris-format,eris-urn]: New fields.
(narinfo-maker): Handle ERIS URN and ERIS format.
* configure.ac: (HAVE_GUILE_ERIS): New conditional.
* gnu/packages/package-management.scm: (guix)[native-inputs]: Add guile-eris.
---
 configure.ac                        |  5 +++++
 gnu/packages/package-management.scm |  1 +
 guix/narinfo.scm                    | 14 ++++++++++----
 guix/scripts/publish.scm            | 24 ++++++++++++++++++++----
 4 files changed, 36 insertions(+), 8 deletions(-)

diff --git a/configure.ac b/configure.ac
index 341cff8fbd..72396be8aa 100644
--- a/configure.ac
+++ b/configure.ac
@@ -170,6 +170,11 @@ GUILE_MODULE_AVAILABLE([have_guile_avahi], [(avahi)])
AM_CONDITIONAL([HAVE_GUILE_AVAHI],
  [test "x$have_guile_avahi" = "xyes"])

dnl Check for Guile-eris.
GUILE_MODULE_AVAILABLE([have_guile_eris], [(eris)])
AM_CONDITIONAL([HAVE_GUILE_ERIS],
  [test "x$have_guile_eris" = "xyes"])

dnl Guile-newt is used by the graphical installer.
GUILE_MODULE_AVAILABLE([have_guile_newt], [(newt)])

diff --git a/gnu/packages/package-management.scm b/gnu/packages/package-management.scm
index 05795824b5..a9094b8b7f 100644
--- a/gnu/packages/package-management.scm
+++ b/gnu/packages/package-management.scm
@@ -404,6 +404,7 @@ (define code
                       ("guile-zstd" ,guile-zstd)
                       ("guile-ssh" ,guile-ssh)
                       ("guile-git" ,guile-git)
                       ("guile-eris" ,guile-eris)

                       ;; XXX: Keep the development inputs here even though
                       ;; they're unnecessary, just so that 'guix environment
diff --git a/guix/narinfo.scm b/guix/narinfo.scm
index 4fc550aa6c..a6a5d3b84b 100644
--- a/guix/narinfo.scm
+++ b/guix/narinfo.scm
@@ -45,6 +45,8 @@ (define-module (guix narinfo)
            narinfo-file-sizes
            narinfo-hash
            narinfo-size
            narinfo-eris-format
            narinfo-eris-urn
            narinfo-references
            narinfo-deriver
            narinfo-system
@@ -68,8 +70,8 @@ (define-module (guix narinfo)

(define-record-type <narinfo>
  (%make-narinfo path uri-base uris compressions file-sizes file-hashes
                 nar-hash nar-size references deriver system
                 signature contents)
                 nar-hash nar-size eris-format eris-urn references deriver
                 system signature contents)
  narinfo?
  (path         narinfo-path)
  (uri-base     narinfo-uri-base)        ;URI of the cache it originates from
@@ -79,6 +81,8 @@ (define-record-type <narinfo>
  (file-hashes  narinfo-file-hashes)
  (nar-hash     narinfo-hash)
  (nar-size     narinfo-size)
  (eris-format  narinfo-eris-format)
  (eris-urn     narinfo-eris-urn)
  (references   narinfo-references)
  (deriver      narinfo-deriver)
  (system       narinfo-system)
@@ -135,7 +139,7 @@ (define (narinfo-maker str cache-url)
  "Return a narinfo constructor for narinfos originating from CACHE-URL.  STR
must contain the original contents of a narinfo file."
  (lambda (path urls compressions file-hashes file-sizes
                nar-hash nar-size references deriver system
                nar-hash nar-size eris-format eris-urn references deriver system
                signature)
    "Return a new <narinfo> object."
    (define len (length urls))
@@ -157,6 +161,8 @@ (define len (length urls))
                     ((lst ...) (map string->number lst)))
                   nar-hash
                   (and=> nar-size string->number)
                   eris-format
                   (if eris-urn (string->uri eris-urn) #f)
                   (string-tokenize references)
                   (match deriver
                     ((or #f "") #f)
@@ -184,7 +190,7 @@ (define* (read-narinfo port #:optional url
                   (narinfo-maker str url)
                   '("StorePath" "URL" "Compression"
                     "FileHash" "FileSize" "NarHash" "NarSize"
                     "References" "Deriver" "System"
                     "ERISFormat" "ERIS" "References" "Deriver" "System"
                     "Signature")
                   '("URL" "Compression" "FileSize" "FileHash"))))

diff --git a/guix/scripts/publish.scm b/guix/scripts/publish.scm
index 6e2b4368da..9c83f5183d 100644
--- a/guix/scripts/publish.scm
+++ b/guix/scripts/publish.scm
@@ -58,6 +58,7 @@ (define-module (guix scripts publish)
  #:use-module (guix workers)
  #:use-module (guix store)
  #:use-module ((guix serialization) #:select (write-file))
  #:use-module (eris)
  #:use-module (zlib)
  #:autoload   (lzlib) (call-with-lzip-output-port
                        make-lzip-output-port)
@@ -146,6 +147,9 @@ (define %default-gzip-compression
  ;; Since we compress on the fly, default to fast compression.
  (compression 'gzip 3))

(define %eris-zstd-compression
  (compression 'zstd 19))

(define (default-compression type)
  (compression type 3))

@@ -324,7 +328,8 @@ (define* (store-item->recutils store-item

(define* (narinfo-string store store-path
                         #:key (compressions (list %no-compression))
                         (nar-path "nar") (file-sizes '()))
                         (nar-path "nar") (file-sizes '())
                         eris-urn)
  "Generate a narinfo key/value string for STORE-PATH; an exception is raised
if STORE-PATH is invalid.  Produce a URL that corresponds to COMPRESSION.  The
narinfo is signed with KEY.  NAR-PATH specifies the prefix for nar URLs.
@@ -347,7 +352,7 @@ (define* (narinfo-string store store-path
StorePath: ~a
~{~a~}\
NarHash: sha256:~a
NarSize: ~d
NarSize: ~d~@[~%ERISFormat: application/x-nix-archive+zstd-19~%ERIS: ~a~]
References: ~a~%"
                             store-path
                             (map (lambda (compression)
@@ -359,7 +364,7 @@ (define* (narinfo-string store store-path
                                                            #:compression
                                                            compression)))
                                  compressions)
                             hash size references))
                             hash size eris-urn references))
         ;; Do not render a "Deriver" line if we are rendering info for a
         ;; derivation.  Also do not render a "System" line that would be
         ;; expensive to compute and is currently unused.
@@ -632,6 +637,16 @@ (define (compressed-nar-size compression)
      (and stat
           (cons compression (stat:size stat)))))

  (define (eris-encode-nar compressions)
    (and (member %eris-zstd-compression compressions)
         (let* ((nar (nar-cache-file cache item
                                     #:compression %eris-zstd-compression))
                (stat (stat nar #f)))
           (and stat
                (call-with-input-file nar
                  (cut eris-encode->string <>
                       #:block-size %eris-block-size-large))))))

  (let ((compression (actual-compressions item compressions)))

    (for-each (cut compress-nar cache item <>) compressions)
@@ -650,7 +665,8 @@ (define (compressed-nar-size compression)
                 (display (narinfo-string store item
                                          #:nar-path nar-path
                                          #:compressions compressions
                                          #:file-sizes sizes)
                                          #:file-sizes sizes
                                          #:eris-urn (eris-encode-nar compression))
                          port)))

             ;; Make the cached narinfo world-readable, contrary to what
-- 
2.34.0
pukkamustard schreef op di 25-01-2022 om 19:21 [+0000]:
pukkamustard schreef op di 25-01-2022 om 19:21 [+0000]:

[RFC PATCH v2 3/5] Add (guix eris). Export this patch

* guix/ipfs.scm: New file.
* Makefile.am (MODULES): Add it.
---
 Makefile.am   |  1 +
 guix/eris.scm | 55 +++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 56 insertions(+)
 create mode 100644 guix/eris.scm

diff --git a/Makefile.am b/Makefile.am
index a10aeb817b..7219386361 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -131,6 +131,7 @@ MODULES =					\
  guix/cve.scm					\
  guix/workers.scm				\
  guix/ipfs.scm					\
  guix/eris.scm					\
  guix/build-system.scm				\
  guix/build-system/android-ndk.scm		\
  guix/build-system/ant.scm			\
diff --git a/guix/eris.scm b/guix/eris.scm
new file mode 100644
index 0000000000..163bbe05ac
--- /dev/null
+++ b/guix/eris.scm
@@ -0,0 +1,55 @@
;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2022 pukkamustard <pukkamustard@posteo.net>
;;;
;;; This file is part of GNU Guix.
;;;
;;; GNU Guix is free software; you can redistribute it and/or modify it
;;; under the terms of the GNU General Public License as published by
;;; the Free Software Foundation; either version 3 of the License, or (at
;;; your option) any later version.
;;;
;;; GNU Guix is distributed in the hope that it will be useful, but
;;; WITHOUT ANY WARRANTY; without even the implied warranty of
;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
;;; GNU General Public License for more details.
;;;
;;; You should have received a copy of the GNU General Public License
;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.

(define-module (guix eris)
  #:use-module (eris)
  #:use-module (eris blocks ipfs)
  #:use-module (web client)
  #:use-module (web response)
  #:use-module (srfi srfi-71)

  #:export (guix-eris-block-reducer))

(define (ipfs-daemon-alive?)
  "Attempt to connect to the IPFS daemon. Returns #t if the daemon is alive
and #f else."
  (with-exception-handler
      (const #f)
    (lambda _
      (let ((response _
                      (http-post (string-append (%ipfs-base-url)
                                                "/api/v0/version"))))
        (equal? 200 (response-code response))))
    #:unwind? #t))

(define guix-eris-block-reducer
  (case-lambda

    ;; Check if IPFS Daemon is running.
    (() (if (ipfs-daemon-alive?)
            (eris-blocks-ipfs-reducer)
            #f))

    ;; Completion. Nothing to do.
    ((_) #t)

    ((ipfs ref-block)
     ;; If IPFS has been initialized store block there
     (if ipfs
         (eris-blocks-ipfs-reducer ipfs ref-block)
         ipfs))))
-- 
2.34.0
pukkamustard schreef op di 25-01-2022 om 19:21 [+0000]:
This (ipfs-daemon-alive?) seems racy, although it's probably not.
Can we do

(define guix-eris-block-reducer
  (case-lambda
    (() (guard (c (oops-it-fails-because-the-daemon-cannot-be-
contacted? c)
                  #false)
          (eris-block-ipfs-reducer))
    [...]))

instead? (I don't think this will work as-is, because from the name and
thunkiness, it would appear that eris-block-ipfs-reducer returns a
procedure ...

Greetings,
Maxime.
pukkamustard schreef op di 25-01-2022 om 19:21 [+0000]:

[RFC PATCH v2 4/5] publish: Add support for storing ERIS encoded blocks to IPFS. Export this patch

* guix/scripts/publish.scm (bake-narinfo+nar): Use guix-eris-block-reducer
from (guix eris).
---
 guix/scripts/publish.scm | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

diff --git a/guix/scripts/publish.scm b/guix/scripts/publish.scm
index 9c83f5183d..556107ab7d 100644
--- a/guix/scripts/publish.scm
+++ b/guix/scripts/publish.scm
@@ -41,6 +41,7 @@ (define-module (guix scripts publish)
  #:use-module (srfi srfi-26)
  #:use-module (srfi srfi-34)
  #:use-module (srfi srfi-37)
  #:use-module (srfi srfi-71)
  #:use-module (web http)
  #:use-module (web request)
  #:use-module (web response)
@@ -58,6 +59,7 @@ (define-module (guix scripts publish)
  #:use-module (guix workers)
  #:use-module (guix store)
  #:use-module ((guix serialization) #:select (write-file))
  #:use-module (guix eris)
  #:use-module (eris)
  #:use-module (zlib)
  #:autoload   (lzlib) (call-with-lzip-output-port
@@ -644,8 +646,14 @@ (define (eris-encode-nar compressions)
                (stat (stat nar #f)))
           (and stat
                (call-with-input-file nar
                  (cut eris-encode->string <>
                       #:block-size %eris-block-size-large))))))
                  (lambda (port)
                    (let ((eris-urn _
                                    (eris-encode port
                                                 #:block-size
                                                 %eris-block-size-large
                                                 #:block-reducer
                                                 guix-eris-block-reducer)))
                      (uri->string eris-urn))))))))

  (let ((compression (actual-compressions item compressions)))

-- 
2.34.0
pukkamustard schreef op di 25-01-2022 om 19:22 [+0000]:

[RFC PATCH v2 5/5] substitute: Fetch substitutes using ERIS. Export this patch

* guix/scripts/substitute.scm (process-substitution): Fetch substitutes using ERIS.
* guix/eris.scm (guix-eris-block-ref): New procedure.
---
 guix/eris.scm               |  7 ++++++-
 guix/scripts/substitute.scm | 21 ++++++++++++++++-----
 2 files changed, 22 insertions(+), 6 deletions(-)

diff --git a/guix/eris.scm b/guix/eris.scm
index 163bbe05ac..0999564c1f 100644
--- a/guix/eris.scm
+++ b/guix/eris.scm
@@ -23,7 +23,8 @@ (define-module (guix eris)
  #:use-module (web response)
  #:use-module (srfi srfi-71)

  #:export (guix-eris-block-reducer))
  #:export (guix-eris-block-reducer
            guix-eris-block-ref))

(define (ipfs-daemon-alive?)
  "Attempt to connect to the IPFS daemon. Returns #t if the daemon is alive
@@ -53,3 +54,7 @@ (define guix-eris-block-reducer
     (if ipfs
         (eris-blocks-ipfs-reducer ipfs ref-block)
         ipfs))))

(define (guix-eris-block-ref ref)
  "Dereference a block for decoding ERIS content"
  (eris-blocks-ipfs-ref ref))
diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm
index 908a8334a8..852264976e 100755
--- a/guix/scripts/substitute.scm
+++ b/guix/scripts/substitute.scm
@@ -62,6 +62,8 @@ (define-module (guix scripts substitute)
  #:use-module (srfi srfi-35)
  #:use-module (web uri)
  #:use-module (guix http-client)
  #:use-module (guix eris)
  #:use-module (eris)
  #:export (%allow-unauthenticated-substitutes?
            %reply-file-descriptor

@@ -486,18 +488,27 @@ (define (fetch uri)
                         #:port port
                         #:keep-alive? #t
                         #:buffered? #f)))))

      (else
       (leave (G_ "unsupported substitute URI scheme: ~a~%")
              (uri->string uri)))))
       (if (and (eris-read-capability? uri))
           (values (eris-decode->port uri
                                      #:block-ref
                                      guix-eris-block-ref) #f)
           (leave (G_ "unsupported substitute URI scheme: ~a~%")
                  (uri->string uri))))))

  (define* (best-uri narinfo #:key (eris? #f))
    (if (and eris? (narinfo-eris-urn narinfo))
        (values (narinfo-eris-urn narinfo) "zstd" #f)
        (narinfo-best-uri narinfo #:fast-decompression?
                          %prefer-fast-decompression?)))

  (unless narinfo
    (leave (G_ "no valid substitute for '~a'~%")
           store-item))

  (let-values (((uri compression file-size)
                (narinfo-best-uri narinfo
                                  #:fast-decompression?
                                  %prefer-fast-decompression?)))
                (best-uri narinfo #:eris? #t)))
    (unless print-build-trace?
      (format (current-error-port)
              (G_ "Downloading ~a...~%") (uri->string uri)))
-- 
2.34.0
pukkamustard schreef op di 25-01-2022 om 19:22 [+0000]:
pukkamustard schreef op di 25-01-2022 om 19:22 [+0000]:
pukkamustard schreef op di 25-01-2022 om 19:22 [+0000]:
pukkamustard schreef op di 25-01-2022 om 19:22 [+0000]: