~abcdw/rde-devel

This thread contains a patchset. You're looking at the original emails, but you may wish to use the patch review UI. Review patch
8 2

[PATCH] gnu: home-services: state: Add home-state-service, git/generic-state

Details
Message ID
<87pmzze9nn.fsf@trop.in>
DKIM signature
missing
Download raw message
Patch: +136 -0
---
This is an initial implementation, patches and comments are welcome.

* How it works
~herd init state~ will create all the neccessary dirs, will clone the
git repos with projects you work on, restore wallpapers dir from backup
server via rsync and so on. That helps at least control and init state
your software depends on, when you switching to new machine for example.

* Future
Maybe later will add some sync/backup and archive capabilities, because
they are important parts of state lifecycle. However, they can be
managed on ad-hoc manner.

gnu/home-services/state.scm | 136 ++++++++++++++++++++++++++++++++++++
 1 file changed, 136 insertions(+)
 create mode 100644 gnu/home-services/state.scm

diff --git a/gnu/home-services/state.scm b/gnu/home-services/state.scm
new file mode 100644
index 0000000..b0f4062
--- /dev/null
+++ b/gnu/home-services/state.scm
@@ -0,0 +1,136 @@
(define-module (gnu home-services state)
  #:use-module (srfi srfi-1)
  #:use-module (ice-9 match)
  #:use-module (gnu home-services)
  #:use-module (gnu home-services-utils)
  #:use-module (gnu home-services shepherd)
  #:use-module (gnu services shepherd)
  #:use-module (gnu services configuration)
  #:use-module (gnu packages ssh)
  #:use-module (guix packages)
  #:use-module (guix gexp)
  #:use-module (guix monads)
  #:use-module (guix modules)
  #:use-module (guix records)

  #:export (home-state-service-type
	    generic-state))


;; TODO: Move to git module? It will use git serializers for
;; generating config and maybe something else.
;; TODO: Add support for custom .git/config
(use-modules (gnu packages version-control))
(define* (git-state path remote #:key (config #f))
  (generic-state
   path
   #:init-gexp
   #~(lambda* (_ self)
       (let* ((meta (car (action self 'metadata)))
	      (path (assoc-ref meta 'path))
	      (remote (assoc-ref meta 'remote)))
	 (format #t "Initializing ~a.\n" self)
	 (flush-all-ports)
	 (let* ((port ((@@ (guix build utils) open-pipe-with-stderr)
		       #$(file-append git "/bin/git") "clone" remote path)))
	   (waitpid WAIT_ANY)
	   (display ((@@ (ice-9 rdelim) read-delimited) "" port))
	   (close-port port))))
   #:additional-metadata `((remote . ,remote))))

;; What about archive/backup/sync actions?
(define* (generic-state
	  path
	  #:key
	  (init-gexp
	   #~(lambda* (_ self)
	       (let ((path (assoc-ref (car (action self 'metadata)) 'path)))
		 (format #t "Initializing ~a.\n" self)
		 (format #t "Creating ~a directory..." path)
		 (mkdir-p path)
		 (display " done"))))
	  (additional-metadata '()))
  "A function which returns a shepherd-service with all required
actions for state management, should be used as a basis for other
state related items like git-state, rsync-state, etc."
  (let ((self (string->symbol
	       (format #f "state-~a" path))))
    (shepherd-service
     (documentation (format #f "Managing state at ~a." path))
     (provision (list self))
     (auto-start? #f)
     (start #~(lambda ()
		(if (car (action '#$self 'state-exists?))
		    #t
		    (begin
		      (format #t "~a is not initilized yet." '#$self)
		      #f))))
     (actions (list
	       (shepherd-action
		(name 'state-exists?)
		(documentation "Check if state file/directory exists.")
		(procedure #~(lambda* (#:rest rest)
			       (file-exists? #$path))))
	       (shepherd-action
		(name 'unchecked-init)
		(documentation "Do not use this action directly.")
		(procedure init-gexp))
	       (shepherd-action
		(name 'metadata)
		(documentation "Returns metadata related to the state.")
		(procedure #~(lambda* (_)
			       (append
				'((path . #$path)
				  (self . #$self))
				'#$additional-metadata))))
	       (shepherd-action
		(name 'init)
		(documentation "Generic initialize.")
		(procedure #~(lambda* (#:rest rest)
			       (if (car (action '#$self 'state-exists?))
				   (format #t "~a already initialized.\n" '#$self)
				   (action '#$self 'unchecked-init '#$self))))))))))

(define (add-shepherd-services services)
  (let* ((service-names
	  (map
	   (lambda (service) (car (shepherd-service-provision service)))
	   services)))
    (append
     services
     (list
      (shepherd-service
       (documentation "Init, update and destroy state.")
       (provision '(state))
       ;; (one-shot? #t)
       (auto-start? #t)
       (start #~(lambda ()
		  (map (lambda (name)
			 (when (car (action name 'state-exists?))
			   (start name)))
		       '#$service-names)))
       (actions (list
		 (shepherd-action
		  (name 'init)
		  (documentation "Initialize all unitialized state- s.")
		  (procedure #~(lambda* (#:rest rest)
				 (map
				  (lambda (name) (action name 'init))
				  '#$service-names)))))))))))

(define home-state-service-type
  (service-type (name 'home-state)
                (extensions
                 (list (service-extension
                        home-shepherd-service-type
                        add-shepherd-services)))
                (default-value
		  (append
		   (map
		    (lambda (path) (generic-state path))
		    (list
		     "/home/bob/work/guix"
		     "/home/bob/work/rde"))
		   (list
		    (git-state "/home/bob/work/talkes" "https://git.savannah.gnu.org/git/guix.git"))))
                (description "A toolset for initializing state.")))
-- 
2.30.1
Details
Message ID
<878s6nt3t1.fsf@yoctocell.xyz>
In-Reply-To
<87pmzze9nn.fsf@trop.in> (view parent)
DKIM signature
pass
Download raw message
This looks really interesting!  Definitely something I would use.

On Mon, Mar 15 2021, Andrew Tropin wrote:

> * How it works
> ~herd init state~ will create all the neccessary dirs, will clone the
> git repos with projects you work on, restore wallpapers dir from backup
> server via rsync and so on. That helps at least control and init state
> your software depends on, when you switching to new machine for example.
>
> * Future
> Maybe later will add some sync/backup and archive capabilities, because
> they are important parts of state lifecycle. However, they can be
> managed on ad-hoc manner.

Should this service only update the state on the initial setup of the
machine or should it also handle updating of the state after the initial
setup has completed, e.g. running `git pull/fetch/push` on some repos or
doing a periodic `rsync`?

> +;; TODO: Move to git module? It will use git serializers for
> +;; generating config and maybe something else.
> +;; TODO: Add support for custom .git/config

Yeah, I think this is a good idea.

> +(use-modules (gnu packages version-control))
> +(define* (git-state path remote #:key (config #f))
> +  (generic-state
> +   path
> +   #:init-gexp
> +   #~(lambda* (_ self)
> +       (let* ((meta (car (action self 'metadata)))
> +	      (path (assoc-ref meta 'path))
> +	      (remote (assoc-ref meta 'remote)))
> +	 (format #t "Initializing ~a.\n" self)
> +	 (flush-all-ports)
> +	 (let* ((port ((@@ (guix build utils) open-pipe-with-stderr)
> +		       #$(file-append git "/bin/git") "clone" remote path)))
> +	   (waitpid WAIT_ANY)
> +	   (display ((@@ (ice-9 rdelim) read-delimited) "" port))
> +	   (close-port port))))
> +   #:additional-metadata `((remote . ,remote))))

Maybe we could use records for configuring the git state?  It would make
it easier to add new options and docs in the future.

> +;; What about archive/backup/sync actions?
> +(define* (generic-state
> +	  path
> +	  #:key
> +	  (init-gexp
> +	   #~(lambda* (_ self)
> +	       (let ((path (assoc-ref (car (action self 'metadata)) 'path)))
> +		 (format #t "Initializing ~a.\n" self)
> +		 (format #t "Creating ~a directory..." path)
> +		 (mkdir-p path)
> +		 (display " done"))))
> +	  (additional-metadata '()))
> +  "A function which returns a shepherd-service with all required
> +actions for state management, should be used as a basis for other
> +state related items like git-state, rsync-state, etc."
> +  (let ((self (string->symbol
> +	       (format #f "state-~a" path))))
> +    (shepherd-service
> +     (documentation (format #f "Managing state at ~a." path))
> +     (provision (list self))
> +     (auto-start? #f)
> +     (start #~(lambda ()
> +		(if (car (action '#$self 'state-exists?))
> +		    #t
> +		    (begin
> +		      (format #t "~a is not initilized yet." '#$self)
> +		      #f))))
> +     (actions (list
> +	       (shepherd-action
> +		(name 'state-exists?)
> +		(documentation "Check if state file/directory exists.")
> +		(procedure #~(lambda* (#:rest rest)
> +			       (file-exists? #$path))))
> +	       (shepherd-action
> +		(name 'unchecked-init)
> +		(documentation "Do not use this action directly.")
> +		(procedure init-gexp))
> +	       (shepherd-action
> +		(name 'metadata)
> +		(documentation "Returns metadata related to the state.")
> +		(procedure #~(lambda* (_)
> +			       (append
> +				'((path . #$path)
> +				  (self . #$self))
> +				'#$additional-metadata))))
> +	       (shepherd-action
> +		(name 'init)
> +		(documentation "Generic initialize.")
> +		(procedure #~(lambda* (#:rest rest)
> +			       (if (car (action '#$self 'state-exists?))
> +				   (format #t "~a already initialized.\n" '#$self)
> +				   (action '#$self 'unchecked-init '#$self))))))))))

I think it would also be cool to add action to list all the
files/directories managed by the state service.

Since there is no way to rollback the state, it should tell the user
which log file to look in to see a history of the actions performed
(created directory, rsynced file, ...).

> +(define (add-shepherd-services services)
> +  (let* ((service-names
> +	  (map
> +	   (lambda (service) (car (shepherd-service-provision service)))
> +	   services)))
> +    (append
> +     services
> +     (list
> +      (shepherd-service
> +       (documentation "Init, update and destroy state.")

You mention update, see the my first comment. :)

Thanks for working on this!
Details
Message ID
<CABrWRW10nPK0HCBWMoyVY6tF00PPXc28bQb-G3TuvRaSM4sAkw@mail.gmail.com>
In-Reply-To
<878s6nt3t1.fsf@yoctocell.xyz> (view parent)
DKIM signature
missing
Download raw message
> This looks really interesting!  Definitely something I would use.
>
> Should this service only update the state on the initial setup of the
> machine or should it also handle updating of the state after the initial
> setup has completed, e.g. running `git pull/fetch/push` on some repos or
> doing a periodic `rsync`?

My orginial idea is to provide a tool for managing the whole lifecycle
of the state. Once you installed all the necessary stuff with `guix
home reconfigure` you are able to run `herd init state` and get all the
folders/files initialized with necessary content. It's a crucial feature
for fast and easy new system setup.

Obviously, the state will be changing over time, for example you are
working on the project and commiting changes/pulling updates and so on.
Such state doesn't need any automated care. Still you want to know which
of your local repos are behind origins and how far, when your pictures
folder was backed up last time and so on. I think it would be cool if
you could do it with `herd status state`. All the information in one
place.

Some state is not intended for ad-hoc syncronization, for example
password-store, mailbox, backups. Such states will be syncronized
manually with `herd sync state` or some mcron jobs. Probably I'll mock
up some implementation for that later.

> > +;; TODO: Move to git module? It will use git serializers for
> > +;; generating config and maybe something else.
> > +;; TODO: Add support for custom .git/config
>
> Yeah, I think this is a good idea.
>
Still tricky one. Probably re-exporting git-state from (gnu
home-services git) will make it relatively convinient to use.

> Maybe we could use records for configuring the git state?  It would make
> it easier to add new options and docs in the future.



> I think it would also be cool to add action to list all the
> files/directories managed by the state service.

If you are talking about just all state-... shepherd services, the list
is avaliable with `herd status`. Started services are initialized,
stopped is not.

> Since there is no way to rollback the state, it should tell the user
> which log file to look in to see a history of the actions performed
> (created directory, rsynced file, ...).

The idea about log files is good. For now it prints everything to
~/.local/var/log/shepherd.log .

> > +      (shepherd-service
> > +       (documentation "Init, update and destroy state.")
>
> You mention update, see the my first comment. :)
>
> Thanks for working on this!

🤙

-- 
Best regards,
Andrew Tropin
Details
Message ID
<CABrWRW3y5hq9+GFQrc2EL7PkDXKxHuZhig37WE94DhuO+oPX8Q@mail.gmail.com>
In-Reply-To
<CABrWRW10nPK0HCBWMoyVY6tF00PPXc28bQb-G3TuvRaSM4sAkw@mail.gmail.com> (view parent)
DKIM signature
missing
Download raw message
Oops, missed that.

> Maybe we could use records for configuring the git state?  It would make
> it easier to add new options and docs in the future.

git-state is just a wrapper around generic-state with more suitable
defaults, don't see how to apply records here to improve it. Share the
code snippet if you have an idea.

-- 
Best regards,
Andrew Tropin
Details
Message ID
<87a6r2rqnu.fsf@yoctocell.xyz>
In-Reply-To
<CABrWRW10nPK0HCBWMoyVY6tF00PPXc28bQb-G3TuvRaSM4sAkw@mail.gmail.com> (view parent)
DKIM signature
pass
Download raw message
On Wed, Mar 17 2021, Andrew Tropin wrote:

>> This looks really interesting!  Definitely something I would use.
>>
>> Should this service only update the state on the initial setup of the
>> machine or should it also handle updating of the state after the initial
>> setup has completed, e.g. running `git pull/fetch/push` on some repos or
>> doing a periodic `rsync`?
>
> My orginial idea is to provide a tool for managing the whole lifecycle
> of the state. Once you installed all the necessary stuff with `guix
> home reconfigure` you are able to run `herd init state` and get all the
> folders/files initialized with necessary content. It's a crucial feature
> for fast and easy new system setup.

Yeah, it is really annoying having to manually git clone a bunch of
repos.

> Obviously, the state will be changing over time, for example you are
> working on the project and commiting changes/pulling updates and so on.
> Such state doesn't need any automated care. Still you want to know which
> of your local repos are behind origins and how far, when your pictures
> folder was backed up last time and so on. I think it would be cool if
> you could do it with `herd status state`. All the information in one
> place.

This would definitely be nice to have.

> Some state is not intended for ad-hoc syncronization, for example
> password-store, mailbox, backups. Such states will be syncronized
> manually with `herd sync state` or some mcron jobs. Probably I'll mock
> up some implementation for that later.

Do you mean that there will be a big state service which takes care of
synchronizing all the things like passwords, mailbox, backups?  Or
should they all be in their own individual service that use mcron to sync
things?

>> I think it would also be cool to add action to list all the
>> files/directories managed by the state service.
>
> If you are talking about just all state-... shepherd services, the list
> is avaliable with `herd status`. Started services are initialized,
> stopped is not.

Ah, ok.

>> Thanks for working on this!
>
> 🤙

I don't know if I am missing some fonts, but my Emacs doesn't render the
emoji thingy. ¯\_(ツ)_/¯
Details
Message ID
<877dm6rqb2.fsf@yoctocell.xyz>
In-Reply-To
<CABrWRW3y5hq9+GFQrc2EL7PkDXKxHuZhig37WE94DhuO+oPX8Q@mail.gmail.com> (view parent)
DKIM signature
pass
Download raw message
On Wed, Mar 17 2021, Andrew Tropin wrote:

>> Maybe we could use records for configuring the git state?  It would make
>> it easier to add new options and docs in the future.
>
> git-state is just a wrapper around generic-state with more suitable
> defaults, don't see how to apply records here to improve it. Share the
> code snippet if you have an idea.

I was thinking about adding an option that would tell it to periodically
do a `git pull/fetch` on a repo, or maybe configure it to only do a
shallow clone.  The user would write:

#+begin_src scheme
(git-state
  (repo-path "/home/alice/src/repo"
  (remote-path "https://git.example.org/repo")
  (extra-options "--shallow")
  (periodic-fetch? #t))
#+end_src

But maybe this gets too complicated and is it out of scope for the state
service.
Details
Message ID
<CABrWRW0BGyuEPf11hyEa9dB9wBCKcDz07501UkxPvyJnGUqZTg@mail.gmail.com>
In-Reply-To
<877dm6rqb2.fsf@yoctocell.xyz> (view parent)
DKIM signature
missing
Download raw message
>> \U0001f919
> I don't know if I am missing some fonts, but my Emacs doesn't render the
> emoji thingy. \u00af\_(\u30c4)_/\u00af

Just a unicode version of Shaka. It renders pretty poorly on my machine too)
https://en.wikipedia.org/wiki/Shaka_sign

> Do you mean that there will be a big state service which takes care of
> synchronizing all the things like passwords, mailbox, backups?  Or
> should they all be in their own individual service that use mcron to sync
> things?

Few small state-/home/user/repo{1,2,3} services, which can be
manipulated by calling some actions on main state service.

> I was thinking about adding an option that would tell it to periodically
> do a `git pull/fetch` on a repo, or maybe configure it to only do a
> shallow clone.  The user would write:

I thought about periodic fetch, it will be a side feature, probably
implemented somewhere in mcron module. Will play a little more with
implementation and will see which one looks best to me.

-- 
Best regards,
Andrew Tropin
Details
Message ID
<87o8fhrg5t.fsf@yoctocell.xyz>
In-Reply-To
<CABrWRW0BGyuEPf11hyEa9dB9wBCKcDz07501UkxPvyJnGUqZTg@mail.gmail.com> (view parent)
DKIM signature
pass
Download raw message
On Wed, Mar 17 2021, Andrew Tropin wrote:
>> Do you mean that there will be a big state service which takes care of
>> synchronizing all the things like passwords, mailbox, backups?  Or
>> should they all be in their own individual service that use mcron to sync
>> things?
>
> Few small state-/home/user/repo{1,2,3} services, which can be
> manipulated by calling some actions on main state service.

Ok, that's cool.

>> I was thinking about adding an option that would tell it to periodically
>> do a `git pull/fetch` on a repo, or maybe configure it to only do a
>> shallow clone.  The user would write:
>
> I thought about periodic fetch, it will be a side feature, probably
> implemented somewhere in mcron module. Will play a little more with
> implementation and will see which one looks best to me.

Since mcron seems to be a service that many other services will depend
on, I think it would make sense to add `home-mcron-service-type` or
something like that.  I will start a new thread for discussing this, WDYT?
Details
Message ID
<CABrWRW1Fq-8mS=MbWJedUpayj1vFg-YE0oNF3zVTYWBMnp29Lg@mail.gmail.com>
In-Reply-To
<87o8fhrg5t.fsf@yoctocell.xyz> (view parent)
DKIM signature
missing
Download raw message
> Since mcron seems to be a service that many other services will depend
> on, I think it would make sense to add `home-mcron-service-type` or
> something like that.  I will start a new thread for discussing this, WDYT?

Sounds good. Yup, start it, please.

I plan to work on state and git home-services next few days, but I will
try to help as much as I can.

-- 
Best regards,
Andrew Tropin
Reply to thread Export thread (mbox)