~sircmpwn/hare-dev

This thread contains a patchset. You're looking at the original emails, but you may wish to use the patch review UI. Review patch
6 4

[PATCH hare] cmd/hare: Print dependencies to a t

Dridi Boukelmoune <dridi.boukelmoune@gmail.com>
Details
Message ID
<20230512175459.32502-1-dridi.boukelmoune@gmail.com>
DKIM signature
missing
Download raw message
Patch: +25 -4
The new hare deps -t option changes the output to a plain text one:

   mod child1
   mod child2
   mod ...

Unlike other output formats, it is not recursive by default, so a
new -r option allows dependencies to also have their dependencies
listed:

   child1::sub other
   child1 child1::sub
   mod child1
   mod child2
   mod ...

The preexisting output formats remain recursive only, leaving the
decision whether or not to change the current behavior outside the
scope of this change.

This format can easily be consumed by field-based programs such as
tsort(1) (to a t!) or the more frequent utilities like sh(1) (while
read loops) or awk(1).

A familiar example:

    hare deps -rt crypto |
    awk '
        BEGIN { print "strict digraph deps {" }
        { print "\t\"" $1 "\" -> \"" $2 "\";" }
        END { print "}" }
    ' |
    dot -T svg >crypto.svg

The motivation is to enable module processing in package managers,
in the non-recursive mode, to derive virtual provides from the first
field and virtual requires from the second one.

Signed-off-by: Dridi Boukelmoune <dridi.boukelmoune@gmail.com>
---
 cmd/hare/deps.ha    | 10 +++++++---
 cmd/hare/main.ha    |  2 ++
 cmd/hare/subcmds.ha | 17 ++++++++++++++++-
 3 files changed, 25 insertions(+), 4 deletions(-)

diff --git a/cmd/hare/deps.ha b/cmd/hare/deps.ha
index f54c9d1e..b29fef43 100644
--- a/cmd/hare/deps.ha
+++ b/cmd/hare/deps.ha
@@ -16,7 +16,8 @@ type depnode = struct {
type dep_cycle = !size;

// depth-first initial exploration, cycle-detection, reverse topological sort
fn explore_deps(ctx: *module::context, stack: *[]str, visited: *[]depnode, ident: str) (size | dep_cycle) = {
fn explore_deps(ctx: *module::context, stack: *[]str, visited: *[]depnode,
		ident: str, recursive: bool) (size | dep_cycle) = {
	// check for cycles
	for (let i = 0z; i < len(stack); i += 1) {
		if (ident == stack[i]) {
@@ -41,8 +42,11 @@ fn explore_deps(ctx: *module::context, stack: *[]str, visited: *[]depnode, ident
	for (let i = 0z; i < len(ver.depends); i += 1) {
		const name = strings::join("::", ver.depends[i]...);
		defer free(name);
		const child = explore_deps(ctx, stack, visited, name)?;
		append(this.depends, child);
		if (recursive) {
			const child = explore_deps(ctx, stack, visited, name,
				true)?;
			append(this.depends, child);
		};
	};
	// reverse-sort depends so that we know the last in the list is the
	// "final" child during show_deps
diff --git a/cmd/hare/main.ha b/cmd/hare/main.ha
index 0164779e..d9140135 100644
--- a/cmd/hare/main.ha
+++ b/cmd/hare/main.ha
@@ -37,6 +37,8 @@ const help: []getopt::help = [
		"prints dependency information for a Hare program",
		('d', "print dot syntax for use with graphviz"),
		('M', "build-dir", "print rules for POSIX make"),
		('r', "print a recursive text output"),
		('t', "print text output for field-based parsing"),
		('T', "tags...", "set build tags"),
		('X', "tags...", "unset build tags"),
		"<path|module>",
diff --git a/cmd/hare/subcmds.ha b/cmd/hare/subcmds.ha
index 7b672765..7ef2849c 100644
--- a/cmd/hare/subcmds.ha
+++ b/cmd/hare/subcmds.ha
@@ -194,6 +194,7 @@ type deps_goal = enum {
	DOT,
	MAKE,
	TERM,
	TEXT,
};

fn deps(cmd: *getopt::command) void = {
@@ -203,6 +204,7 @@ fn deps(cmd: *getopt::command) void = {

	let build_dir: str = "";
	let goal = deps_goal::TERM;
	let text_recursive = false;
	for (let i = 0z; i < len(cmd.opts); i += 1) {
		let opt = cmd.opts[i];
		switch (opt.0) {
@@ -211,6 +213,10 @@ fn deps(cmd: *getopt::command) void = {
		case 'M' =>
			goal = deps_goal::MAKE;
			build_dir = opt.1;
		case 'r' =>
			text_recursive = true;
		case 't' =>
			goal = deps_goal::TEXT;
		case 'T' =>
			tags = match (addtags(tags, opt.1)) {
			case void =>
@@ -270,7 +276,9 @@ fn deps(cmd: *getopt::command) void = {
	for (let i = 0z; i < len(ver.depends); i += 1) {
		const name = strings::join("::", ver.depends[i]...);
		defer free(name);
		const child = match (explore_deps(&ctx, &stack, &visited, name)) {
		let recursive = goal != deps_goal::TEXT || text_recursive;
		const child = match (explore_deps(&ctx, &stack, &visited, name,
				recursive)) {
		case let index: size => yield index;
		case let start: dep_cycle =>
			const chain = strings::join(" -> ", stack[start..]...);
@@ -300,6 +308,13 @@ fn deps(cmd: *getopt::command) void = {
			};
		};
		fmt::println("}")!;
	case deps_goal::TEXT =>
		for (let i = 0z; i < len(visited); i += 1) {
			for (let j = 0z; j < len(visited[i].depends); j += 1) {
				const child = visited[visited[i].depends[j]];
				fmt::printfln("{} {}", visited[i].ident, child.ident)!;
			};
		};
	case deps_goal::MAKE =>
		abort("-M option not implemented yet");
	};
-- 
2.40.1

[hare/patches] build success

builds.sr.ht <builds@sr.ht>
Details
Message ID
<CSKHRVCIW5F1.26ADHHNBEAD35@cirno2>
In-Reply-To
<20230512175459.32502-1-dridi.boukelmoune@gmail.com> (view parent)
DKIM signature
missing
Download raw message
hare/patches: SUCCESS in 1m40s

[cmd/hare: Print dependencies to a t][0] from [Dridi Boukelmoune][1]

[0]: https://lists.sr.ht/~sircmpwn/hare-dev/patches/41087
[1]: dridi.boukelmoune@gmail.com

✓ #988824 SUCCESS hare/patches/alpine.yml  https://builds.sr.ht/~sircmpwn/job/988824
✓ #988825 SUCCESS hare/patches/freebsd.yml https://builds.sr.ht/~sircmpwn/job/988825
Details
Message ID
<0b994335-5e89-3c6d-b37f-6edb9742533d@posteo.net>
In-Reply-To
<20230512175459.32502-1-dridi.boukelmoune@gmail.com> (view parent)
DKIM signature
missing
Download raw message
> The new hare deps -t option changes the output to a plain text one:
> 
>     mod child1
>     mod child2
>     mod ...
i think this is a useful thing to have, this is a good idea <3

> Unlike other output formats, it is not recursive by default, so a
> new -r option allows dependencies to also have their dependencies
> listed:
> 
>     child1::sub other
>     child1 child1::sub
>     mod child1
>     mod child2
>     mod ...
imo it should be recursive by default, and the user can just filter on 
the first field if they don't want all the recursive information.

> This format can easily be consumed by field-based programs such as
> tsort(1) (to a t!) or the more frequent utilities like sh(1) (while
> read loops) or awk(1).
> 
> A familiar example:
> 
>      hare deps -rt crypto |
>      awk '
>          BEGIN { print "strict digraph deps {" }
>          { print "\t\"" $1 "\" -> \"" $2 "\";" }
>          END { print "}" }
>      ' |
>      dot -T svg >crypto.svg
> 
> The motivation is to enable module processing in package managers,
> in the non-recursive mode, to derive virtual provides from the first
> field and virtual requires from the second one.
jsyk, the information i've quoted here could go into the send-email 
annotation, rather than the commit message (so that it's just recorded 
as your motivation to patch reviewers). the way to do this is, write the 
rest of the stuff in the commit message, and then when you do git 
send-email, write

	git send-email --annotate ...

and it'll show you your patch, and then:
> Signed-off-by: Dridi Boukelmoune <dridi.boukelmoune@gmail.com>
> ---below ^ that dashed line, you can write the rest of the annotations. See 
here: https://git-send-email.io/#step-4

side note: if you print it out right, the tsort shouldn't be necessary, 
since the exploration function sorts in reverse topological order anyway 
(so you can print in topological order by iterating in reverse).

additionaly: beware that the -T option (build tags) has the possibility 
to alter the list of dependencies that are printed (because some modules 
only rely on others for testing, or under libc, etc. so you probably 
actually want to create a completely custom function if you would like 
to know an exhaustive list of modules that a certain module may depend 
on. suggestions for this in a moment...

>   // depth-first initial exploration, cycle-detection, reverse topological sort
> -fn explore_deps(ctx: *module::context, stack: *[]str, visited: *[]depnode, ident: str) (size | dep_cycle) = {
> +fn explore_deps(ctx: *module::context, stack: *[]str, visited: *[]depnode,
> +		ident: str, recursive: bool) (size | dep_cycle) = {
unfortunately this function is going to be deleted in the rewrite of the 
build driver and hare::module that i am currently working on. my 
suggestion is to wait until the rewrite has been merged, and then try 
using the dependency-scanning functions that will be included there in 
order to write your custom function.

~Autumn
Dridi Boukelmoune <dridi.boukelmoune@gmail.com>
Details
Message ID
<CAJYncsi83ECEfJfPBwfesN2H4F68JrPAemW5dR+t3now6z9ZBA@mail.gmail.com>
In-Reply-To
<0b994335-5e89-3c6d-b37f-6edb9742533d@posteo.net> (view parent)
DKIM signature
missing
Download raw message
On Fri, May 12, 2023 at 6:37 PM Autumn! <autumnull@posteo.net> wrote:
>
> > The new hare deps -t option changes the output to a plain text one:
> >
> >     mod child1
> >     mod child2
> >     mod ...
> i think this is a useful thing to have, this is a good idea <3
>
> > Unlike other output formats, it is not recursive by default, so a
> > new -r option allows dependencies to also have their dependencies
> > listed:
> >
> >     child1::sub other
> >     child1 child1::sub
> >     mod child1
> >     mod child2
> >     mod ...
> imo it should be recursive by default, and the user can just filter on
> the first field if they don't want all the recursive information.

I think this is a more subtle problem than it looks. Because of the
module filesystem layout in hare, recursive may have two meanings.

See for example:

    $ hare deps -t /usr/src/hare/stdlib/crypto
    crypto crypto::argon2
    crypto io
    crypto errors
    crypto endian
    crypto crypto::math
    crypto crypto::mac
    crypto crypto::poly1305
    crypto crypto::cipher
    crypto crypto::chacha
    crypto bufio
    crypto bytes
    $ hare deps -t /usr/src/hare/stdlib/crypto/argon2
    argon2 types
    argon2 io
    argon2 hash
    argon2 errors
    argon2 endian
    argon2 crypto::math
    argon2 crypto::blake2b
    argon2 bytes
    argon2 bufio

Ideally, I should see crypto::argon2 for the second one, but because
I'm querying a directory I think this caveat is justified.

But if I wanted to point `hare deps` at a top-level directory (for
example in my package's build root!) and recursively list a module's
dependency and its sub-modules, then filtering with the first column
defeats the purpose of leveraging the hare build system. If I have to
translate this knowledge in downstream packaging I'm running an
increasing risk of getting my automatic package dependencies right.

My next patch would have introduced a -s option to recursively list
sub-module dependencies.

Example:

    $ hare deps -st /usr/src/hare/stdlib/crypto
    crypto::argon2 types
    crypto::argon2 io
    crypto::argon2 hash
    crypto::argon2 errors
    crypto::argon2 endian
    crypto::argon2 crypto::math
    crypto::argon2 crypto::blake2b
    crypto::argon2 bytes
    crypto::argon2 bufio
    [...]
    crypto crypto::argon2
    crypto io
    crypto errors
    crypto endian
    crypto crypto::math
    crypto crypto::mac
    crypto crypto::poly1305
    crypto crypto::cipher
    crypto crypto::chacha
    crypto bufio
    crypto bytes

So I'm making the distinction here between recursively listing
sub-modules and transitive dependencies. In general, I find it
familiar enough to require a -r option to list things recursively, and
in this context I felt -r applied best to dependencies.

<snip>
> jsyk, the information i've quoted here could go into the send-email
> annotation, rather than the commit message (so that it's just recorded
> as your motivation to patch reviewers). the way to do this is, write the
> rest of the stuff in the commit message, and then when you do git
> send-email, write
>
>         git send-email --annotate ...
>
> and it'll show you your patch, and then:
> > Signed-off-by: Dridi Boukelmoune <dridi.boukelmoune@gmail.com>
> > ---below ^ that dashed line, you can write the rest of the annotations. See
> here: https://git-send-email.io/#step-4

I got the same tip on a previous patch, but felt that I would rather
have this in the actual commit log.

> side note: if you print it out right, the tsort shouldn't be necessary,
> since the exploration function sorts in reverse topological order anyway
> (so you can print in topological order by iterating in reverse).

I'm not so much interested in passing this output to tsort(1), so my
commit messages probably doesn't properly convey that I wanted to
adopt tsort(1)'s input format.

> additionaly: beware that the -T option (build tags) has the possibility
> to alter the list of dependencies that are printed (because some modules
> only rely on others for testing, or under libc, etc. so you probably
> actually want to create a completely custom function if you would like
> to know an exhaustive list of modules that a certain module may depend
> on. suggestions for this in a moment...

I had not considered that, thanks.

> >   // depth-first initial exploration, cycle-detection, reverse topological sort
> > -fn explore_deps(ctx: *module::context, stack: *[]str, visited: *[]depnode, ident: str) (size | dep_cycle) = {
> > +fn explore_deps(ctx: *module::context, stack: *[]str, visited: *[]depnode,
> > +             ident: str, recursive: bool) (size | dep_cycle) = {
> unfortunately this function is going to be deleted in the rewrite of the
> build driver and hare::module that i am currently working on. my
> suggestion is to wait until the rewrite has been merged, and then try
> using the dependency-scanning functions that will be included there in
> order to write your custom function.

That was my very first hare patch, and I'm sure I will throw many more
away. The thought of trying to share logic between the -d and -t
options crossed my mind but it felt overkill. Rewriting my tiny patch
when the dependency scanner is ready is perfectly fine. Please either
CC me on the relevant thread or update this thread when it's ready.

Thanks for the initial feedback!

Dridi
Details
Message ID
<CVBZT3X57HTW.2TOYJPHE1BDUQ@monch>
In-Reply-To
<CAJYncsi83ECEfJfPBwfesN2H4F68JrPAemW5dR+t3now6z9ZBA@mail.gmail.com> (view parent)
DKIM signature
missing
Download raw message
On Sat May 13, 2023 at 8:56 PM UTC, Dridi Boukelmoune wrote:
> That was my very first hare patch, and I'm sure I will throw many more
> away. The thought of trying to share logic between the -d and -t
> options crossed my mind but it felt overkill. Rewriting my tiny patch
> when the dependency scanner is ready is perfectly fine. Please either
> CC me on the relevant thread or update this thread when it's ready.

just wanted to let you know that module3 has now been merged :)
Dridi Boukelmoune <dridi.boukelmoune@gmail.com>
Details
Message ID
<CAJYncshDRigPcXFd5tJCpzXSy7hW333mk+pLnc2c6qeX80fbKQ@mail.gmail.com>
In-Reply-To
<CVBZT3X57HTW.2TOYJPHE1BDUQ@monch> (view parent)
DKIM signature
missing
Download raw message
On Wed, Sep 6, 2023 at 4:58 PM Ember Sawady <ecs@d2evs.net> wrote:
>
> On Sat May 13, 2023 at 8:56 PM UTC, Dridi Boukelmoune wrote:
> > That was my very first hare patch, and I'm sure I will throw many more
> > away. The thought of trying to share logic between the -d and -t
> > options crossed my mind but it felt overkill. Rewriting my tiny patch
> > when the dependency scanner is ready is perfectly fine. Please either
> > CC me on the relevant thread or update this thread when it's ready.
>
> just wanted to let you know that module3 has now been merged :)

Thanks!

Not sure when I will revisit this, but I will.

It has become a little inconvenient to follow hare development lately.
I maintain my own harec and hare packages locally, but now it's
relying on unreleased qbe features. I will shave the qbe yak first and
adjust my workflow.

Dridi
Details
Message ID
<CVC53122G37G.11QQLEVY9DMHK@monch>
In-Reply-To
<CAJYncshDRigPcXFd5tJCpzXSy7hW333mk+pLnc2c6qeX80fbKQ@mail.gmail.com> (view parent)
DKIM signature
missing
Download raw message
On Wed Sep 6, 2023 at 9:04 PM UTC, Dridi Boukelmoune wrote:
> It has become a little inconvenient to follow hare development lately.
> I maintain my own harec and hare packages locally, but now it's
> relying on unreleased qbe features. I will shave the qbe yak first and
> adjust my workflow.

yeah, you could ask ~mpu/qbe@lists.sr.ht for a release
Reply to thread Export thread (mbox)