Hey all,

Since (per the meeting notes) we're doing LXD for the actual runtimes
and we want unattended upgrades, I looked into it a bit. I think that
we may not want to do live-patching after all because it quite
literally prevents us from scaling (3 servers are free, from the 4th
it's $225 per year per server.)

Outside of this it seems like the path for automatic kernel upgrades is
to generate our own patch files and apply them manually through our
automation.


...in light of this and since we intend to be tolerant of a single dead
server anyway, what do we think about just having the servers reboot
when they get a new kernel? We'd just configure the prod servers to run
their unattended upgrades 12 hours apart or similar. Another option is
to do unattended kernel upgrades but only reboot when we manually tell
it do to so.

In other news, I've created a private repo for me to experiment with
mage in. Private because with deploy infrastructure there may be some
secrets or non-reusable info stored in there. Let me know if you'd like
access, since sourcehut doesn't yet have teams.

> Since (per the meeting notes) we're doing LXD for the actual runtimes
> and we want unattended upgrades, I looked into it a bit. I think that
> we may not want to do live-patching after all because it quite
> literally prevents us from scaling (3 servers are free, from the 4th
> it's $225 per year per server.)

Oof. Yeah, we can't afford that crap. I guess we'll need to have some
kind of scheduled reboot cadence?

> Outside of this it seems like the path for automatic kernel upgrades is
> to generate our own patch files and apply them manually through our
> automation.

I think we can start with just taking reboots. it's much easier to
implement/understand. Doing our own live patching would be cool, but
we don't need to start there.

> ...in light of this and since we intend to be tolerant of a single dead
> server anyway, what do we think about just having the servers reboot
> when they get a new kernel? We'd just configure the prod servers to run
> their unattended upgrades 12 hours apart or similar. Another option is
> to do unattended kernel upgrades but only reboot when we manually tell
> it do to so.

I probably should have read the whole email before responding to parts
of it, huh? Yeah, let's just schedule reboots at something like 8 hour
intervals.

> In other news, I've created a private repo for me to experiment with
> mage in. Private because with deploy infrastructure there may be some
> secrets or non-reusable info stored in there. Let me know if you'd like
> access, since sourcehut doesn't yet have teams.

I'd really like to find a way to not store secrets in the git repo, or
at least to encrypt them. PGP and git-crypt are both solid options
here. What kind of secrets do you expect the repo to contain?

Thanks for working on this! For my part, I've been playing with LXD
some locally. I'm impressed by the documented features of the
clustering, but I haven't yet been able to try them. I hope they work
as well as the box says they should ;P.

Cheers,
Chris