~abcdw/rde-discuss

8 3

Issues with Native Comp; feature-emacs is not used as a builder

Details
Message ID
<87h6gm5ll4.fsf@samuelculpepper.com>
DKIM signature
pass
Download raw message
Hi RDE,

Having some trouble with native compilation; simply put, it seems that
the home-service does not respect the emacs package specified in the
user-configuration, and does not propagate package-rewites under any circumstances.

Context:
- Running on :: HEAD=3344ce422f1c3d6f3e43815bbca76992a0aa8e0a,
- feature-emacs :: supplied with arg =#:emacs (@ (rde packages emacs) emacs-next-pgtk-latest)=
- unabridged configuration :: https://pastebin.com/EhbsNa2X

I can hardly grok the code in =src/rde/home/services/emacs.scm=, but
various parts of emacs have been broken (relating to incorrect bytecode)
since pulling in the following changes from 3-6 months ago:

- 01a2767c3ae056959941f37ee1df2e6c12cdd9cd (+home-emacs-extension{s}, -home-elisp-feature)
- ce43c3e3b5b3cf471770ed207345dbd17f8cee9d (elisp-packages-rewrites as service-arg)
- d596b18bf4a96d58c159c65738c5c8142f20cb26 (home-emacs-configuration-{emacs, elisp-packages-rewrites})
- edb2313333598a37035055eab396a9c4265baeea (rewrite)
- eff7e53c4118e5ff434f64ccb1a12e7bf46823e8 (rebuild-elisp-packages? -> native-comp?)
- 2a9b4f59170504db80add1f3ab1fd5bb2a96fb2c (home-emacs-configuration-package)

Even given the extensive rewrites, it seems that =emacs-minimal= (version 29.3) is
used instead of =home-emacs-configuration-emacs= (30.0.50).

This is further supported by =#:native-comp?= not having any effect when
supplied with #t or #f, or when enumerating =#:emacs= through other
versions of packages.

We can observe the bytecode version in a collection of =.elc= files,
taken from packages in the guix-home profile, as follows:

$ guix package -p $HOME/.guix-home/profile -I |
    awk '{print $4}' |
    parallel "find {} -name '*.elc'" |
    parallel 'echo -n {}; sed -ne 3p {}' |
    sort | uniq |
    rg -o 'Emacs version.+' |
    sort | uniq -c

- 1608 Emacs version 29.3
- 767 Emacs version 30.0.50

---

Without comprehending the necessary service/feature code, nor the
package-rewrite spec, it is hard to know if this is a library-error or a
user-error.

The ordering of features in the RDE config is important, yet mysterious,
so this issue may arise due to a data-race with respect to the specified
emacs package.

Will update here if I can find some solutions; will take another go with
home-emacs-extension and the package-rewrite spec.


--
Best regards,
Samuel Culpepper
Details
Message ID
<87a5mdldhr.fsf@ngraves.fr>
In-Reply-To
<87h6gm5ll4.fsf@samuelculpepper.com> (view parent)
DKIM signature
missing
Download raw message
On 2024-03-31 19:59, Samuel Culpepper wrote:

> Hi RDE,
>
> Having some trouble with native compilation; simply put, it seems that
> the home-service does not respect the emacs package specified in the
> user-configuration, and does not propagate package-rewites under any circumstances.
>
> Context:
> - Running on :: HEAD=3344ce422f1c3d6f3e43815bbca76992a0aa8e0a,
> - feature-emacs :: supplied with arg =#:emacs (@ (rde packages emacs) emacs-next-pgtk-latest)=
> - unabridged configuration :: https://pastebin.com/EhbsNa2X
>
> I can hardly grok the code in =src/rde/home/services/emacs.scm=, but
> various parts of emacs have been broken (relating to incorrect bytecode)
> since pulling in the following changes from 3-6 months ago:
>
> - 01a2767c3ae056959941f37ee1df2e6c12cdd9cd (+home-emacs-extension{s}, -home-elisp-feature)
> - ce43c3e3b5b3cf471770ed207345dbd17f8cee9d (elisp-packages-rewrites as service-arg)
> - d596b18bf4a96d58c159c65738c5c8142f20cb26 (home-emacs-configuration-{emacs, elisp-packages-rewrites})
> - edb2313333598a37035055eab396a9c4265baeea (rewrite)
> - eff7e53c4118e5ff434f64ccb1a12e7bf46823e8 (rebuild-elisp-packages? -> native-comp?)
> - 2a9b4f59170504db80add1f3ab1fd5bb2a96fb2c (home-emacs-configuration-package)
>
> Even given the extensive rewrites, it seems that =emacs-minimal= (version 29.3) is
> used instead of =home-emacs-configuration-emacs= (30.0.50).
>
> This is further supported by =#:native-comp?= not having any effect when
> supplied with #t or #f, or when enumerating =#:emacs= through other
> versions of packages.
>
> We can observe the bytecode version in a collection of =.elc= files,
> taken from packages in the guix-home profile, as follows:
>
> $ guix package -p $HOME/.guix-home/profile -I |
>     awk '{print $4}' |
>     parallel "find {} -name '*.elc'" |
>     parallel 'echo -n {}; sed -ne 3p {}' |
>     sort | uniq |
>     rg -o 'Emacs version.+' |
>     sort | uniq -c
>
> - 1608 Emacs version 29.3
> - 767 Emacs version 30.0.50
>
> ---
>
> Without comprehending the necessary service/feature code, nor the
> package-rewrite spec, it is hard to know if this is a library-error or a
> user-error.
>
> The ordering of features in the RDE config is important, yet mysterious,
> so this issue may arise due to a data-race with respect to the specified
> emacs package.
>
> Will update here if I can find some solutions; will take another go with
> home-emacs-extension and the package-rewrite spec.
>

Hi Samuel,

Thanks for reporting this. On my side, I have

   1139 Emacs version 29.3
   1666 Emacs version 30.0.50

and I do have changes when I change the #:emacs component, although
indeed not all packages are affected. Will try to debug this now.
>
> --
> Best regards,
> Samuel Culpepper

-- 
Best regards,
Nicolas Graves
Details
Message ID
<877chhkvz9.fsf@ngraves.fr>
In-Reply-To
<87a5mdldhr.fsf@ngraves.fr> (view parent)
DKIM signature
missing
Download raw message
On 2024-04-01 09:59, Nicolas Graves wrote:

> On 2024-03-31 19:59, Samuel Culpepper wrote:
>
>> Without comprehending the necessary service/feature code, nor the
>> package-rewrite spec, it is hard to know if this is a library-error or a
>> user-error.
>>
>> The ordering of features in the RDE config is important, yet mysterious,
>> so this issue may arise due to a data-race with respect to the specified
>> emacs package.
>>
>> Will update here if I can find some solutions; will take another go with
>> home-emacs-extension and the package-rewrite spec.
>>
>
> Hi Samuel,
>
> Thanks for reporting this. On my side, I have
>
>    1139 Emacs version 29.3
>    1666 Emacs version 30.0.50
>
> and I do have changes when I change the #:emacs component, although
> indeed not all packages are affected. Will try to debug this now.

I'm currently rebuilding my packages and sent a patch in 
https://lists.sr.ht/~abcdw/rde-devel/patches/50634

The code isn't actually wrong, but it did forgot a lot of cases in
package-replacement, the first of which is emacs-minimal.

I guess the rest of the conversation will happen on the patch thread,
don't hesitate to test it also.

-- 
Best regards,
Nicolas Graves
Details
Message ID
<87jzlgw7xf.fsf@trop.in>
In-Reply-To
<87h6gm5ll4.fsf@samuelculpepper.com> (view parent)
DKIM signature
pass
Download raw message
On 2024-03-31 19:59, Samuel Culpepper wrote:

> Hi RDE,
>

Hey Samuel, nice to see you!

Nicolas is already replied, I will just add a few minor comments.

> Having some trouble with native compilation; simply put, it seems that
> the home-service does not respect the emacs package specified in the
> user-configuration, and does not propagate package-rewites under any circumstances.
>
> Context:
> - Running on :: HEAD=3344ce422f1c3d6f3e43815bbca76992a0aa8e0a,
> - feature-emacs :: supplied with arg =#:emacs (@ (rde packages emacs) emacs-next-pgtk-latest)=

Deprecated emacs-next-pgtk-* in favor of emacs-pgtk.

> - unabridged configuration :: https://pastebin.com/EhbsNa2X
>
> I can hardly grok the code in =src/rde/home/services/emacs.scm=, but
> various parts of emacs have been broken (relating to incorrect bytecode)
> since pulling in the following changes from 3-6 months ago:
>
> - 01a2767c3ae056959941f37ee1df2e6c12cdd9cd (+home-emacs-extension{s}, -home-elisp-feature)
> - ce43c3e3b5b3cf471770ed207345dbd17f8cee9d (elisp-packages-rewrites as service-arg)
> - d596b18bf4a96d58c159c65738c5c8142f20cb26 (home-emacs-configuration-{emacs, elisp-packages-rewrites})
> - edb2313333598a37035055eab396a9c4265baeea (rewrite)
> - eff7e53c4118e5ff434f64ccb1a12e7bf46823e8 (rebuild-elisp-packages? -> native-comp?)
> - 2a9b4f59170504db80add1f3ab1fd5bb2a96fb2c (home-emacs-configuration-package)
>
> Even given the extensive rewrites, it seems that =emacs-minimal= (version 29.3) is
> used instead of =home-emacs-configuration-emacs= (30.0.50).
>
> This is further supported by =#:native-comp?= not having any effect when
> supplied with #t or #f, or when enumerating =#:emacs= through other
> versions of packages.

native comp is still WIP in guix, so it make take a few more months to
start working properly.

>
> We can observe the bytecode version in a collection of =.elc= files,
> taken from packages in the guix-home profile, as follows:
>
> $ guix package -p $HOME/.guix-home/profile -I |
>     awk '{print $4}' |
>     parallel "find {} -name '*.elc'" |
>     parallel 'echo -n {}; sed -ne 3p {}' |
>     sort | uniq |
>     rg -o 'Emacs version.+' |
>     sort | uniq -c
>
> - 1608 Emacs version 29.3
> - 767 Emacs version 30.0.50
>
> ---
>
> Without comprehending the necessary service/feature code, nor the
> package-rewrite spec, it is hard to know if this is a library-error or a
> user-error.
>
> The ordering of features in the RDE config is important, yet mysterious,
> so this issue may arise due to a data-race with respect to the specified
> emacs package.

While ordering of features can affect some minor things a bit, but in
this case it should not be related.

>
> Will update here if I can find some solutions; will take another go with
> home-emacs-extension and the package-rewrite spec.

Thank you, keep us posted! :)

-- 
Best regards,
Andrew Tropin
Details
Message ID
<87bk6qepk1.fsf@samuelculpepper.com>
In-Reply-To
<87jzlgw7xf.fsf@trop.in> (view parent)
DKIM signature
pass
Download raw message
👋 Hey Andrew, Nicolas,

Thank you both for responding with such detail and speed.

After separating my config from my fork of RDE, I am clueless as to how
I can locally build (and I still have not learned how to debug guile properly),
which precludes my further contributions.

Might you direct me here?  Sorry to be a help-vampire, but will need some momentum!
- i.e. "use guix channels pointing to a local repo,
        and do X to ensure .guix-authorizations is valid"
- i.e. "make a pre-inst-env from a local repo"

Would love to work on the emacs-native-comp, bring my python/cpp/rust
development features to RDE, and generally be better
developer-user-operator of this lovely software environment!

---

Thanks again both,
  Samuel
Date: Wed, 03 Apr 2024 11:57:35 +0200
Message-ID: <87frw2epkg.fsf@samuelculpepper.com>
Details
Message ID
<87h6giei1p.fsf@ngraves.fr>
In-Reply-To
<87bk6qepk1.fsf@samuelculpepper.com> (view parent)
DKIM signature
missing
Download raw message
On 2024-04-03 11:57, Samuel Culpepper wrote:

> 👋 Hey Andrew, Nicolas,
>
> Thank you both for responding with such detail and speed.
>
> After separating my config from my fork of RDE, I am clueless as to how
> I can locally build (and I still have not learned how to debug guile properly),
> which precludes my further contributions.
>
> Might you direct me here?  Sorry to be a help-vampire, but will need some momentum!
> - i.e. "use guix channels pointing to a local repo,
>         and do X to ensure .guix-authorizations is valid"
> - i.e. "make a pre-inst-env from a local repo"
>
> Would love to work on the emacs-native-comp, bring my python/cpp/rust
> development features to RDE, and generally be better
> developer-user-operator of this lovely software environment!

There are indeed various approaches. You can take a look at the various
1.7. People's rde configurations (in the README).

I would probably recommend Miguel Angel Moreno's dotfiles here, but all
approaches revolve around the same ideas.

The one tricky point is to properly configure your load-paths, you can
find a ./pre-inst-env in Miguel's dotfiles that does it, Andrew does it
in a Makefile and I do it in guile directly. But all do indicate where
the repositories (guix, rde...) actually are if I'm not mistaken.

Andrew will probably have more insights. Maybe we could also make a
blank template configuration in a repo with presets that people can fork
to make their own config from there, Andrew WDYT ?

>
> ---
>
> Thanks again both,
>   Samuel
> Date: Wed, 03 Apr 2024 11:57:35 +0200
> Message-ID: <87frw2epkg.fsf@samuelculpepper.com>

-- 
Best regards,
Nicolas Graves
Details
Message ID
<87frvy9131.fsf@samuelculpepper.com>
In-Reply-To
<87h6giei1p.fsf@ngraves.fr> (view parent)
DKIM signature
pass
Download raw message
> take a look at the various 1.7. People's rde configurations [...]
> The one tricky point is to properly configure your load-paths, [...]
> [...] all do indicate where the repositories (guix, rde...) actually are
Great advice.  I have read through the example configs, and landed back
using a pre-inst-env.

The most simple update is now an update is a matter of:
    #+begin_src
    git submodule update --recursive --remote \
      && guix pull -C channels/local.scm
    #+end_src

where channels/local.scm is:
    #+begin_quote
    ;; -*- mode: scheme -*-
    (use-modules (guix ci)
                 (guix channels))
    (list
     (channel
      (name 'guix)
      (url (format #f "file://~a/guix" (getenv "PWD")))
      (branch "master")
      (introduction
       (make-channel-introduction
        "9edb3f66fd807b096b48283debdcddccfea34bad"
        (openpgp-fingerprint
         "BBB0 2DDF 2CEA F6A8 0D1D  E643 A2A0 6DF2 A33A 54FA"))))
         ;;; ...
         )
    #+end_quote     

Some reflections on this local-tree approach, and general decoupling:

- 1.  It strikes me that local channels work well be simple git submodules,
  rather than arbitrary paths on the system, such that the whole RDE
  environment can be recursively cloned & built using only fully
  qualified paths from inside the repo.

- 2.  Pinned updates (a la Andrew's =target/profiles/guix= + inferiors) are
  a great way to have minimal (thus, quick) updates to the build
  environment, helped along with =guix.lock= and =guix-local.lock=.

- 3.  Update footprint; further decoupling, of =base-packages=, into
  profiles / manifests, which are pinned to the same inferiors but separate
  revisions as to the configs. (I feel this pain when large packages or dependency-graphs/toolchains
  have updates which get in the way of simple config updates).

- 4.  =guix [home|system] build= needs a =--root= argument, to avoid
  GCing the build dependencies.  Maybe I can work around this by some
  direct =guix build= invocation, but it escapes me at this time (i.e
  unsure how to "package" my config)


Regarding #3, here is an example of what I'm trying to achieve,
expressed as minimal updates:

(init)
- build  :: target/profiles/guix :: rev0
- pkg0   :: build@0              :: rev0
- home   :: build@0,pkg0@0       :: rev0
- system :: build@0,pkg0@0       :: rev0

(core update)
- build  :: target/profiles/guix :: rev0,1    (update all channels)
- pkg0   :: build@0              :: rev0      (no update)
- home   :: build@1,pkg0@0       :: rev0,1    (update core)
- system :: build@1,pkg0@0       :: rev0,1    (update core)

(pkg update)
- build  :: target/profiles/guix :: rev0,1    (no update)
- pkg0   :: build@1              :: rev0,1    (update all pkg)
- home   :: build@1,pkg0@0       :: rev0,1    (no update)
- system :: build@1,pkg0@0       :: rev0,1    (no update)

(pkg-pin update)
- build  :: target/profiles/guix :: rev0,1    (no update)
- pkg0   :: build@1              :: rev0,1    (no update)
- home   :: build@1,pkg0@1       :: rev0,1,2  (update pins to pkg)
- system :: build@1,pkg0@1       :: rev0,1,2  (update pins to pkg)

Note that (core update) is what becomes blocked by pkg updates, and is
the target for update-cost reduction.

---

Will clean up my configs, and share the source in this thread later.
Thanks for all the help!

Samuel
Details
Message ID
<87bk6m8sru.fsf@samuelculpepper.com>
In-Reply-To
<87frvy9131.fsf@samuelculpepper.com> (view parent)
DKIM signature
pass
Download raw message
Samuel Culpepper <samuel@samuelculpepper.com> writes:
> Will clean up my configs, and share the source in this thread later.
> Thanks for all the help!

Find the procedure here -- the local master of each submodule/channel is
used by the pinned inferior build profiles:

- update channel ::
  https://github.com/qzdl/.files/blob/f590dee/Makefile#L265
- update inferiors ::
  https://github.com/qzdl/.files/blob/f590dee/Makefile#L271 
- reconf local, basically nicked from Andrew ::
  https://github.com/qzdl/.files/blob/f590dee/Makefile#L52

Reconf, etc with:
$ make -B guix-local && make reconfigure/local/ixy
Details
Message ID
<87il0sascd.fsf@trop.in>
In-Reply-To
<87frvy9131.fsf@samuelculpepper.com> (view parent)
DKIM signature
pass
Download raw message
On 2024-04-06 13:34, Samuel Culpepper wrote:

>> take a look at the various 1.7. People's rde configurations [...]
>> The one tricky point is to properly configure your load-paths, [...]
>> [...] all do indicate where the repositories (guix, rde...) actually are
> Great advice.  I have read through the example configs, and landed back
> using a pre-inst-env.
>
> The most simple update is now an update is a matter of:
>     #+begin_src
>     git submodule update --recursive --remote \
>       && guix pull -C channels/local.scm
>     #+end_src
>
> where channels/local.scm is:
>     #+begin_quote
>     ;; -*- mode: scheme -*-
>     (use-modules (guix ci)
>                  (guix channels))
>     (list
>      (channel
>       (name 'guix)
>       (url (format #f "file://~a/guix" (getenv "PWD")))
>       (branch "master")
>       (introduction
>        (make-channel-introduction
>         "9edb3f66fd807b096b48283debdcddccfea34bad"
>         (openpgp-fingerprint
>          "BBB0 2DDF 2CEA F6A8 0D1D  E643 A2A0 6DF2 A33A 54FA"))))
>          ;;; ...
>          )
>     #+end_quote     
>
> Some reflections on this local-tree approach, and general decoupling:
>
> - 1.  It strikes me that local channels work well be simple git submodules,
>   rather than arbitrary paths on the system, such that the whole RDE
>   environment can be recursively cloned & built using only fully
>   qualified paths from inside the repo.
>
> - 2.  Pinned updates (a la Andrew's =target/profiles/guix= + inferiors) are
>   a great way to have minimal (thus, quick) updates to the build
>   environment, helped along with =guix.lock= and =guix-local.lock=.
>
> - 3.  Update footprint; further decoupling, of =base-packages=, into
>   profiles / manifests, which are pinned to the same inferiors but separate
>   revisions as to the configs. (I feel this pain when large packages or dependency-graphs/toolchains
>   have updates which get in the way of simple config updates).
>
> - 4.  =guix [home|system] build= needs a =--root= argument, to avoid
>   GCing the build dependencies.  Maybe I can work around this by some
>   direct =guix build= invocation, but it escapes me at this time (i.e
>   unsure how to "package" my config)
>
>
> Regarding #3, here is an example of what I'm trying to achieve,
> expressed as minimal updates:
>
> (init)
> - build  :: target/profiles/guix :: rev0
> - pkg0   :: build@0              :: rev0
> - home   :: build@0,pkg0@0       :: rev0
> - system :: build@0,pkg0@0       :: rev0
>
> (core update)
> - build  :: target/profiles/guix :: rev0,1    (update all channels)
> - pkg0   :: build@0              :: rev0      (no update)
> - home   :: build@1,pkg0@0       :: rev0,1    (update core)
> - system :: build@1,pkg0@0       :: rev0,1    (update core)
>
> (pkg update)
> - build  :: target/profiles/guix :: rev0,1    (no update)
> - pkg0   :: build@1              :: rev0,1    (update all pkg)
> - home   :: build@1,pkg0@0       :: rev0,1    (no update)
> - system :: build@1,pkg0@0       :: rev0,1    (no update)
>
> (pkg-pin update)
> - build  :: target/profiles/guix :: rev0,1    (no update)
> - pkg0   :: build@1              :: rev0,1    (no update)
> - home   :: build@1,pkg0@1       :: rev0,1,2  (update pins to pkg)
> - system :: build@1,pkg0@1       :: rev0,1,2  (update pins to pkg)
>
> Note that (core update) is what becomes blocked by pkg updates, and is
> the target for update-cost reduction.
>
> ---
>
> Will clean up my configs, and share the source in this thread later.
> Thanks for all the help!


Hey Samuel!

I feel your pain.  We did a guix-related contract recently and the
easiest solution was:
https://github.com/abcdw/notes/blob/fce3dc1/notes/20240210123238-2024_02_10_guix_workflow.org#L5

However, it very far from perfect.  Tons of Makefile code, slow and
irreproducible guix pull. https://issues.guix.gnu.org/69284#5

In addition to that the guix tooling itself is pretty limited at the
moment.  There is no tool for making template projects, like
https://github.com/seancorfield/clj-new 

It's not possible to build clj-new like tool and sanely interact with
guile projects, because there is no lightweight Guile package and
load-path manager like https://clojure.org/reference/deps_edn

That being said we don't have tools to provide a good UX and initial
setup for RDE at the moment.  We slowly building tools, but not yet
here.


It would be cool to have a reproducible channels first and some API for
building such. See POTENTIAL IMPROVEMENT 2.
https://github.com/abcdw/notes/blob/fce3dc1/notes/20240210123238-2024_02_10_guix_workflow.org#exact-guix-environment

After that deps.edn like tool for guile can help to provide convinient
interface to interact with Guile-based APIs and get rid of guix cli,
guix pull, Makefiles. And get experience similiar to nix flakes.

On top of this thing a tool for project templates can be implemented.



P.S. I would advice not to use inferiors, because I suspect that it's a
very hacky mechanism and can break in many various unexpected ways.

-- 
Best regards,
Andrew Tropin
Reply to thread Export thread (mbox)