Hey - wanted to get some thoughts on something,
Currently, seed configuration is stored in a flat file with the format:
```
<peer>@<addr>
<peer>@<addr>
etc.
```
This works fine for simple use-cases, but our use-case is a bit more complex:
our seeds have more than one component, eg. they have a git bridge and an http
api. Each of these may be configured differently. Another difference is that
we allow users to configure per-working-copy seeds that overwrite the "global"
config. These are stored in the local git configuration.
To have the flexibility to configure all of this, I've been thinking of storing
things in git config in the following format:
[rad.seed.hybiuzf4onqwszanr47qbd9g6ok7ypghtaq7rp9cey561oxtbksn7c]
p2p = rad://hybiuzf4onqwszanr47qbd9g6ok7ypghtaq7rp9cey561oxtbksn7c@127.0.0.1:8776
git = http://127.0.0.1:8778
api = http://127.0.0.1:8777
This would allow the user to configure individual endpoints. It also can be
overwritten in working copies if desired. For the "global" config, it would
go in the monorepo's git/config.
Obviously, we want to have compatibility with what link is doing, eg. the 'seeds'
file, but we also need this flexibility. Thoughts?
-a
On 12/07/22 07:53am, Alexis Sellier wrote:
> Hey - wanted to get some thoughts on something,>> Currently, seed configuration is stored in a flat file with the format:>> ```> <peer>@<addr>> <peer>@<addr>> etc.> ```>> This works fine for simple use-cases, but our use-case is a bit more complex:> our seeds have more than one component, eg. they have a git bridge and an http> api. Each of these may be configured differently. Another difference is that> we allow users to configure per-working-copy seeds that overwrite the "global"> config. These are stored in the local git configuration.>> To have the flexibility to configure all of this, I've been thinking of storing> things in git config in the following format:>> [rad.seed.hybiuzf4onqwszanr47qbd9g6ok7ypghtaq7rp9cey561oxtbksn7c]> p2p = rad://hybiuzf4onqwszanr47qbd9g6ok7ypghtaq7rp9cey561oxtbksn7c@127.0.0.1:8776> git = http://127.0.0.1:8778> api = http://127.0.0.1:8777
This doesn't feel like it really belongs in git config. Would it not
make more sense to have a `rad.toml` or similar?
> This would allow the user to configure individual endpoints. It also can be> overwritten in working copies if desired. For the "global" config, it would> go in the monorepo's git/config.>> Obviously, we want to have compatibility with what link is doing, eg. the 'seeds'> file, but we also need this flexibility. Thoughts?
I think from our perspective the main change we would make here would be
to make sure that anywhere we read from the seeds file you can also
override those seeds with your own. That way higher level `rad` tooling
can load seeds (and other configuration) from wherever it wants and pass
that through.
One thing to note is that here's no way to pass working copy specific
information through to the other side of the `gitd`. You _could_ use
`--server-option` (once we've implemented the parsing of such things)
but that feels very clunky. I don't think this would be a problem for
you though because you would just run the `gitd` without `--push-seeds`
and `--fetch-seeds` so there is no interaction with the seed except
explicit `rad sync`s.
------- Original Message -------
On Tuesday, July 12th, 2022 at 12:08, Alex Good <alex@memoryandthought.me> wrote:
> > > On 12/07/22 07:53am, Alexis Sellier wrote:> > > Hey - wanted to get some thoughts on something,> > > > Currently, seed configuration is stored in a flat file with the format:> > > > `<peer>@<addr> <peer>@<addr> etc.`> > > > This works fine for simple use-cases, but our use-case is a bit more complex:> > our seeds have more than one component, eg. they have a git bridge and an http> > api. Each of these may be configured differently. Another difference is that> > we allow users to configure per-working-copy seeds that overwrite the "global"> > config. These are stored in the local git configuration.> > > > To have the flexibility to configure all of this, I've been thinking of storing> > things in git config in the following format:> > > > [rad.seed.hybiuzf4onqwszanr47qbd9g6ok7ypghtaq7rp9cey561oxtbksn7c]> > p2p = rad://hybiuzf4onqwszanr47qbd9g6ok7ypghtaq7rp9cey561oxtbksn7c@127.0.0.1:8776> > git = http://127.0.0.1:8778> > api = http://127.0.0.1:8777> > > This doesn't feel like it really belongs in git config. Would it not> make more sense to have a `rad.toml` or similar?>
Yeah, I think the advantage of using git here is being able to use the `git config`
commands as well as the scoping system (local takes precedence over global),
but there's one interesting advantage of using a `rad.toml` file: you
could check it in to the source tree. I'll ponder on this.
> > This would allow the user to configure individual endpoints. It also can be> > overwritten in working copies if desired. For the "global" config, it would> > go in the monorepo's git/config.> > > > Obviously, we want to have compatibility with what link is doing, eg. the 'seeds'> > file, but we also need this flexibility. Thoughts?> > > I think from our perspective the main change we would make here would be> to make sure that anywhere we read from the seeds file you can also> override those seeds with your own. That way higher level `rad` tooling> can load seeds (and other configuration) from wherever it wants and pass> that through.>
Yeah, though if you mean that the lower-level tooling needs to support
other configuration scopes, I'm not sure that's a necessity: seeds can
always be overriden via command-line flags or library options.
> One thing to note is that here's no way to pass working copy specific> information through to the other side of the `gitd`. You could use> `--server-option` (once we've implemented the parsing of such things)> but that feels very clunky. I don't think this would be a problem for> you though because you would just run the `gitd` without `--push-seeds`> and `--fetch-seeds` so there is no interaction with the seed except> explicit `rad sync`s.
Yeah. I think there's a limitation once you start to use `git` over `rad`.
I'm not sure we should solve for this -- supporting default seeds via
gitd is probably good enough. If we really wanted this though (per-project
seeds with gitd support), I guess we could have all that config centralized
in the global seed config, eg. via `[<urn>.seed]` namespaces.
Locally you'd have just `[seed]`, and globally you'd have both `[seed]`
and (optional) `[<urn>.seed]` sections.
Little update on this. I'm going to trial the following system, loosely modeled after cargo:
# File format
For now, something like this:
```
[[seed]]
name = "my-seed"
p2p = "rad://hyb5to4rshftx4apgmu9s6wnsp4ddmp1mz6ijh4qqey7fb8wrpawxa@pine.radicle.garden:8776"
git = "https://pine.radicle.garden:443"
api = "https://pine.radicle.garden:8777"
[[seed]]
...
```
# Profiles
Each profile can have a `config.toml` file inside the profile folder. Seeds specified
in here are used accross all projects. This is modeled after `~/.cargo/config.toml`
# Projects
Projects can check-in a `Radicle.toml` file in their source tree which takes
precedence over the profile config when commands are run from a working copy.
This is modeled after `Cargo.toml`.
-a
------- Original Message -------
On Tuesday, July 12th, 2022 at 16:31, Alexis Sellier <alexis@radicle.foundation> wrote:
> > > > > > > > ------- Original Message -------> On Tuesday, July 12th, 2022 at 12:08, Alex Good alex@memoryandthought.me wrote:> > > > > On 12/07/22 07:53am, Alexis Sellier wrote:> > > > > Hey - wanted to get some thoughts on something,> > > > > > Currently, seed configuration is stored in a flat file with the format:> > > > > > `<peer>@<addr> <peer>@<addr> etc.`> > > > > > This works fine for simple use-cases, but our use-case is a bit more complex:> > > our seeds have more than one component, eg. they have a git bridge and an http> > > api. Each of these may be configured differently. Another difference is that> > > we allow users to configure per-working-copy seeds that overwrite the "global"> > > config. These are stored in the local git configuration.> > > > > > To have the flexibility to configure all of this, I've been thinking of storing> > > things in git config in the following format:> > > > > > [rad.seed.hybiuzf4onqwszanr47qbd9g6ok7ypghtaq7rp9cey561oxtbksn7c]> > > p2p = rad://hybiuzf4onqwszanr47qbd9g6ok7ypghtaq7rp9cey561oxtbksn7c@127.0.0.1:8776> > > git = http://127.0.0.1:8778> > > api = http://127.0.0.1:8777> > > > This doesn't feel like it really belongs in git config. Would it not> > make more sense to have a `rad.toml` or similar?> > > Yeah, I think the advantage of using git here is being able to use the `git config`> commands as well as the scoping system (local takes precedence over global),> but there's one interesting advantage of using a `rad.toml` file: you> could check it in to the source tree. I'll ponder on this.> > > > This would allow the user to configure individual endpoints. It also can be> > > overwritten in working copies if desired. For the "global" config, it would> > > go in the monorepo's git/config.> > > > > > Obviously, we want to have compatibility with what link is doing, eg. the 'seeds'> > > file, but we also need this flexibility. Thoughts?> > > > I think from our perspective the main change we would make here would be> > to make sure that anywhere we read from the seeds file you can also> > override those seeds with your own. That way higher level `rad` tooling> > can load seeds (and other configuration) from wherever it wants and pass> > that through.> > > Yeah, though if you mean that the lower-level tooling needs to support> other configuration scopes, I'm not sure that's a necessity: seeds can> always be overriden via command-line flags or library options.> > > One thing to note is that here's no way to pass working copy specific> > information through to the other side of the `gitd`. You could use> > `--server-option` (once we've implemented the parsing of such things)> > but that feels very clunky. I don't think this would be a problem for> > you though because you would just run the `gitd` without `--push-seeds`> > and `--fetch-seeds` so there is no interaction with the seed except> > explicit `rad sync`s.> > > Yeah. I think there's a limitation once you start to use `git` over `rad`.> I'm not sure we should solve for this -- supporting default seeds via> gitd is probably good enough. If we really wanted this though (per-project> seeds with gitd support), I guess we could have all that config centralized> in the global seed config, eg. via `[<urn>.seed]` namespaces.> > > Locally you'd have just `[seed]`, and globally you'd have both `[seed]`> and (optional) `[<urn>.seed]` sections.
On 14/07/22 08:54am, Alexis Sellier wrote:
> Little update on this. I'm going to trial the following system, loosely modeled after cargo:>> # File format>> For now, something like this:> ```> [[seed]]> name = "my-seed"> p2p = "rad://hyb5to4rshftx4apgmu9s6wnsp4ddmp1mz6ijh4qqey7fb8wrpawxa@pine.radicle.garden:8776"> git = "https://pine.radicle.garden:443"> api = "https://pine.radicle.garden:8777"
An alternative to all of these might be to use SRV records? Or support
some kind of service discovery API on the server itself?
>> [[seed]]> ...> ```>> # Profiles>> Each profile can have a `config.toml` file inside the profile folder. Seeds specified> in here are used accross all projects. This is modeled after `~/.cargo/config.toml`>
I'm generally wary of using toml if we don't have to because it's more
complicated for CLI tools to interact with. The current `profile/seeds`
file is very straightforward for CLI to interact with as it's a very
simple line based format. Is the TOML worth its weight here?
> # Projects>> Projects can check-in a `Radicle.toml` file in their source tree which takes> precedence over the profile config when commands are run from a working copy.> This is modeled after `Cargo.toml`.
Checking this in seems like a bad idea to me. I can see how it's
convenient for delegates who are all working together using the same
seed, but for any other scenario it's quite inconvenient. For example,
non-delegate contributors would need to modify the config file to point
at their own seed and then maintain that change alongside upstream
changes.
More broadly I think we should be aiming for seeds to be scoped to the
user rather than the project - this provides a path for seeds to be
viable businesses because the _user_ pays the seed for connectivity and
storage rather than the project (most of which will be open source and
thus not have an easy way to pay for things). Furthermore, the current
required direct configuration of a seed for each URN is not the design
we want, it's just what's working right now, in future we want the seed
location to be a dynamic property that clients can look up. Checking in
these configs creates a de facto project scoped seed which I think could
be hard to change in future. I think we would be better served by a
`.gitinored` `.radicle` file.
> > [[seed]]> > name = "my-seed"> > p2p = "rad://hyb5to4rshftx4apgmu9s6wnsp4ddmp1mz6ijh4qqey7fb8wrpawxa@pine.radicle.garden:8776"> > git = "https://pine.radicle.garden:443"> > api = "https://pine.radicle.garden:8777"> > > An alternative to all of these might be to use SRV records? Or support> some kind of service discovery API on the server itself?>
Yeah, though that is much harder for users to configure and only works with DNS.
It would be nice if it was supported in the future though, as long as it wasn't
the only way. It's also very tricky to manage DNS record changes without a
defacto "admin".
> > I'm generally wary of using toml if we don't have to because it's more> complicated for CLI tools to interact with. The current `profile/seeds`> file is very straightforward for CLI to interact with as it's a very> simple line based format. Is the TOML worth its weight here?>
This is one of the trade-offs with not using git-config. An alternative
would be to use JSON, which can be parsed with eg. `jq` from the CLI.
I don't have any strong preference between JSON and TOML, although one
is easier to edit by humans (TOML) and the other is perhaps easier for
machines. The only reason TOML is an acceptable choice at all is that our
tools and libraries are written in Rust, so it feels pretty native.
I don't really see a viable option with a flat seed file, there are a bunch
of other settings I can already think of that users may want, and probably
more that I haven't thought of. Examples are per-peer seeds, pull vs. push seeds
as well as configuration unrelated to seeds, such as command aliases.
> > # Projects> > > > Projects can check-in a `Radicle.toml` file in their source tree which takes> > precedence over the profile config when commands are run from a working copy.> > This is modeled after `Cargo.toml`.> > > Checking this in seems like a bad idea to me. I can see how it's> convenient for delegates who are all working together using the same> seed, but for any other scenario it's quite inconvenient. For example,> non-delegate contributors would need to modify the config file to point> at their own seed and then maintain that change alongside upstream> changes.>
Checking in is optional of course, but the above is only an issue if
the seed doesn't allow anyone but the delegates to push to it.
> More broadly I think we should be aiming for seeds to be scoped to the> user rather than the project - this provides a path for seeds to be> viable businesses because the user pays the seed for connectivity and> storage rather than the project (most of which will be open source and> thus not have an easy way to pay for things). Furthermore, the current> required direct configuration of a seed for each URN is not the design> we want, it's just what's working right now, in future we want the seed> location to be a dynamic property that clients can look up. Checking in> these configs creates a de facto project scoped seed which I think could> be hard to change in future. I think we would be better served by a> `.gitinored` `.radicle` file.
I agree in an ideal world the seed is dynamic and discovered on sync. But
I can think of a lot of cases where this isn't what you want, or simply
doesn't work. If a user/community/org/project wants to host their code
on a stable address, they should be able to do that, and guarantee better
data availability for their projects that way. If that stable address is
down, only then would the client need to ask the network, which will always
be slower.
Rather than ignoring the file though, it should be possible to make
the tools work around the issues you mention above. It's
better to have the information there and be able to ignore it or
bypass it, than to not have the information at all. This is why
my preference would be to have this information checked in. If a
configuration file such as this didn't exist, I would consider
adding my seeds to the README or something, for example, to make
sure people know about it, if they got the repo from github or some
other location.
---
If we can't find common ground here, I'm also open to keep supporting
the 'seeds' flat-file as a fall-back fwiw; though it won't work for
all seed deployments, eg. some users have exposed their link node
on a different sub-domain than their http-api due to technical
constraints.