Hello all o/
I did a lot of this work about a year ago but never actually finished it up and
sent it. I didn't actually make any commits while I was working so I spent a
couple hours today breaking things up and trying to order it all as logically as
possible.
I'm happy to make changes and submit further patchsets if anything needs
modification.
Cheers \o
*** BLURB HERE ***
Amolith (12):
Ignore compiled binaries
Build relay from source & clean up temp dirs
Reorganise directories and embed configs
Move setup source, add templating & more configs
Add Prometheus and Caddy to Apt target
Add Prometheus and Caddy targets
Disallow incoming, allow relay/client connections
Add copy function for writing from embedded fs
Add arbor target for setting the relay up
Use new build target syntax
Small changes to User target & normalising logs
Add go.mod and go.sum
.gitignore | 6 +-
deploy.sh | 36 +++--
deploy/setup.go | 12 --
files/Caddyfile | 3 +
files/arbor-relay.service | 11 ++
files/prometheus.yml | 47 +++++++
{deploy/files => files}/sshd_config | 0
{deploy/files => files}/sudoers | 0
go.mod | 5 +
go.sum | 4 +
setup.go | 200 +++++++++++++++++++++++++---
11 files changed, 282 insertions(+), 42 deletions(-)
mode change 100644 => 100755 deploy.sh
delete mode 100644 deploy/setup.go
create mode 100644 files/Caddyfile
create mode 100644 files/arbor-relay.service
create mode 100644 files/prometheus.yml
rename {deploy/files => files}/sshd_config (100%)
rename {deploy/files => files}/sudoers (100%)
create mode 100644 go.mod
create mode 100644 go.sum
--
2.37.1
Do we want to allow ssh in too?
You already allowed SSH in the original code you wrote. https://git.sr.ht/~tekk/arbor-infra/tree/main/item/setup.go#L98-101
Should this include hte service name somewhere in the sh.Run invocation? I don't run systemd so I can't test off-hand.
No, any time you modify or add systemd services, you have to run daemon-reload to re-create dependency trees and regenerate things. After modyfing or adding services, you can't start, stop, enable, or disable anything until you run daemon-reload.
Daniel Wilkins <tekk@linuxmail.org> writes:
Copy & paste the following snippet into your terminal to import this patchset into git:
curl -s https://lists.sr.ht/~whereswaldon/arbor-dev/patches/34067/mbox | git am -3Learn more about email & git
Signed-off-by: Amolith <amolith@secluded.site> --- .gitignore | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/.gitignore b/.gitignore index cbf226f..98dce49 100644 --- a/.gitignore +++ b/.gitignore @@ -1,2 +1,6 @@ out -*~ \ No newline at end of file +*~ + +# Ignores the compiled binaries +mage +files/relay -- 2.37.1
Signed-off-by: Amolith <amolith@secluded.site> --- deploy.sh | 36 ++++++++++++++++++++++++------------ 1 file changed, 24 insertions(+), 12 deletions(-) mode change 100644 => 100755 deploy.sh diff --git a/deploy.sh b/deploy.sh old mode 100644 new mode 100755 index 9532400..14e9f9b --- a/deploy.sh +++ b/deploy.sh @@ -1,33 +1,45 @@ #!/bin/sh set -e -function usage { - echo $0 [ip] +usage() { + echo "$0 [server.ip.address]" } if [[ "$1" == "" ]]; then - usage; exit 1; + usage + exit 1 fi echo "NOTE: In order to run this script your ssh key must have root access to the instance." echo "Your machine must also have mage installed and configured so that the bootstrap magefile can be built." echo "Press enter to continue" -read junk +read -r junk -WORKDIR=$(dirname $0) +WORKDIR=$(dirname "$0") RETURN=$(pwd) -echo "Building mage image" -cd $PWD +TEMPDIR=$(mktemp -d) -mage -compile deploy/mage +echo "Building relay binary" -tar czvf mage.tar.gz deploy +curl --tlsv1.2 -o "$TEMPDIR/gover" https://git.sr.ht/~whereswaldon/gover/blob/main/gover +chmod +x "$TEMPDIR/gover" +PREFIX=$TEMPDIR/ "$TEMPDIR/gover" -scp mage.tar.gz "root@$1:" +git clone --depth=1 https://git.sr.ht/~whereswaldon/sprout-go "$TEMPDIR/sprout-go" +cd "$TEMPDIR/sprout-go/cmd/relay" && "$TEMPDIR/bin/go" build && mv -v ./relay "$WORKDIR/files/relay" -ssh "root@$1" tar xzvf mage.tar.gz +echo "Building mage binary" +cd "$WORKDIR" +mage -compile mage -ssh "root@$1" sh -c "pwd && cd deploy && ./mage deploy" +scp mage "root@$1:" + +ssh root@"$1" sh -c "./mage deploy" + +cd "$RETURN" +rm -rf "$TEMPDIR" + +echo "Deploy finished!" -- 2.37.1
Signed-off-by: Amolith <amolith@secluded.site>
---
{deploy/files => files}/sshd_config | 0
{deploy/files => files}/sudoers | 0
setup.go | 5 +++++
3 files changed, 5 insertions(+)
rename {deploy/files => files}/sshd_config (100%)
rename {deploy/files => files}/sudoers (100%)
diff --git a/deploy/files/sshd_config b/files/sshd_config
similarity index 100%
rename from deploy/files/sshd_config
rename to files/sshd_config
diff --git a/deploy/files/sudoers b/files/sudoers
similarity index 100%
rename from deploy/files/sudoers
rename to files/sudoers
diff --git a/setup.go b/setup.go
index 8880136..de63f25 100644
--- a/setup.go
+++ b/setup.go
@@ -3,12 +3,17 @@
package main
import (
+ "embed"
"fmt"
"github.com/magefile/mage/mg"
"github.com/magefile/mage/sh"
)
+// Import configuration templates
+//go:embed files
+var files embed.FS
+
// Handles initial configuration of the instance
func Deploy() error {
mg.Deps(Users, Sudoers, Apt, Firewall)
--
2.37.1
Signed-off-by: Amolith <amolith@secluded.site> --- deploy/setup.go | 12 ---------- files/Caddyfile | 3 +++ files/arbor-relay.service | 11 +++++++++ files/prometheus.yml | 47 +++++++++++++++++++++++++++++++++++++++ setup.go | 39 ++++++++++++++++++++++++++++++++ 5 files changed, 100 insertions(+), 12 deletions(-) delete mode 100644 deploy/setup.go create mode 100644 files/Caddyfile create mode 100644 files/arbor-relay.service create mode 100644 files/prometheus.yml diff --git a/deploy/setup.go b/deploy/setup.go deleted file mode 100644 index bbb2af9..0000000 --- a/deploy/setup.go @@ -1,12 +0,0 @@ -//+build mage - -package main - -import ( - "github.com/magefile/mage/sh" -) - -// says hoi >:3 -func Build() error { - return sh.Run("/bin/echo", "hoooooooooi!") -} diff --git a/files/Caddyfile b/files/Caddyfile new file mode 100644 index 0000000..7b790f9 --- /dev/null +++ b/files/Caddyfile @@ -0,0 +1,3 @@ +{{ .Hostname }} { + reverse_proxy localhost:9090 +} diff --git a/files/arbor-relay.service b/files/arbor-relay.service new file mode 100644 index 0000000..49e281b --- /dev/null +++ b/files/arbor-relay.service @@ -0,0 +1,11 @@ +[Unit] +Description=Arbor Chat Relay + +[Service] +User=arbor-relay +Group=arbor-relay +ExecStart=/usr/local/bin/relay --grovepath /usr/local/share/arbor-relay --tls-ip 0.0.0.0 --tls-port 7117 --keypath /var/lib/caddy/.local/share/caddy/certificates/acme-v02.api.letsencrypt.org-directory/{{ .Hostname }}/{{ .Hostname }}.key --certpath /var/lib/caddy/.local/share/caddy/certificates/acme-v02.api.letsencrypt.org-directory/{{ .Hostname }}/{{ .Hostname }}.crt arbor.chat:7117 +Restart=Always + +[Install] +WantedBy=multi-user.target diff --git a/files/prometheus.yml b/files/prometheus.yml new file mode 100644 index 0000000..2e011d6 --- /dev/null +++ b/files/prometheus.yml @@ -0,0 +1,47 @@ +global: + scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute. + evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute. + # scrape_timeout is set to the global default (10s). + + # Attach these labels to any time series or alerts when communicating with + # external systems (federation, remote storage, Alertmanager). + external_labels: + monitor: '{{ .Hostname }}' + +# Alertmanager configuration +alerting: + alertmanagers: + - static_configs: + - targets: ['localhost:9093'] + +# Load rules once and periodically evaluate them according to the global 'evaluation_interval'. +rule_files: + # - "first_rules.yml" + # - "second_rules.yml" + +# A scrape configuration containing exactly one endpoint to scrape: +# Here it's Prometheus itself. +scrape_configs: + # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config. + - job_name: 'prometheus' + + # Override the global default and scrape targets from this job every 5 seconds. + scrape_interval: 5s + scrape_timeout: 5s + + # metrics_path defaults to '/metrics' + # scheme defaults to 'http'. + + static_configs: + - targets: ['localhost:9090'] + + - job_name: node + # If prometheus-node-exporter is installed, grab stats about the local + # machine by default. + static_configs: + - targets: ['localhost:9100'] + + - job_name: arbor-relay + static_configs: + - targets: ['localhost:2112'] + diff --git a/setup.go b/setup.go index de63f25..5fcfb18 100644 --- a/setup.go +++ b/setup.go @@ -5,6 +5,10 @@ package main import ( "embed" "fmt" + "log" + "os" + "strings" + "text/template" "github.com/magefile/mage/mg" "github.com/magefile/mage/sh" @@ -113,3 +117,38 @@ func Firewall() error { return sh.Run("ufw", "enable") } + +// config() takes source template and destination file arguments and executes +// the template replacing `hostname` with the system's hostname. +func config(rfile string, wfile string) error { + log.Println("Generating config from", rfile, "and writing to", wfile) + + // Create a new template using the rfile (read file) + data, err := files.ReadFile(rfile) + tmpl := template.Must(template.New("temp").Parse(string(data))) + + hostname, err := os.ReadFile("/etc/hostname") + if err != nil { + return err + } + + // Open wfile (write file) for writing generated config + file, err := os.OpenFile(wfile, os.O_CREATE|os.O_RDWR, 0660) + if err != nil { + return err + } + + // Compose config file using the template, substituting Hostname with the + // contents of /etc/hostname and trimming all whitespace + err = tmpl.Execute(file, map[string]string{"Hostname": strings.TrimSpace(string(hostname))}) + if err != nil { + return err + } + + err = file.Close() + if err != nil { + return err + } + + return nil +} -- 2.37.1
Signed-off-by: Amolith <amolith@secluded.site> --- setup.go | 46 ++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 42 insertions(+), 4 deletions(-) diff --git a/setup.go b/setup.go index 5fcfb18..1c1c602 100644 --- a/setup.go +++ b/setup.go @@ -5,8 +5,11 @@ package main import ( "embed" "fmt" + "io" "log" + "net/http" "os" + "os/exec" "strings" "text/template" @@ -74,13 +77,48 @@ func SshConfig() error { } func Apt() error { - fmt.Println("Running target apt") - var apt = sh.RunCmd("env", "DEBIAN_FRONTEND=noninteractive", "apt-get", "-qq") + log.Println("Running target apt") + apt := sh.RunCmd("env", "DEBIAN_FRONTEND=noninteractive", "apt-get", "-qq") apt("update") + // Install requisite packages for adding Caddy's PPA + apt("install", "debian-keyring", "debian-archive-keyring", "apt-transport-https") - // Safe to run an install if a package is already installed. - return apt("install", "golang-1.14") + resp, err := http.Get("https://dl.cloudsmith.io/public/caddy/stable/gpg.key") + if err != nil { + return err + } + + if resp.StatusCode != http.StatusOK { + return fmt.Errorf("Unexpected response code: %d", resp.StatusCode) + } + + command := exec.Command("apt-key", "add", "-") + command.Stdin = resp.Body + err = command.Run() + if err != nil { + return err + } + + resp, err = http.Get("https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt") + if err != nil { + return err + } + if resp.StatusCode != http.StatusOK { + return fmt.Errorf("unexpected response code: %d", resp.StatusCode) + } + command = exec.Command("tee", "/etc/apt/sources.list.d/caddy-stable.list") + command.Stdin = resp.Body + err = command.Run() + if err != nil { + return err + } + + // Update before installing packages + apt("update") + + // Install necessary packages + return apt("install", "prometheus", "git", "acl", "caddy") } func Firewall() error { -- 2.37.1
Signed-off-by: Amolith <amolith@secluded.site> --- setup.go | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) diff --git a/setup.go b/setup.go index 1c1c602..e9f4cbc 100644 --- a/setup.go +++ b/setup.go @@ -23,8 +23,8 @@ var files embed.FS // Handles initial configuration of the instance func Deploy() error { - mg.Deps(Users, Sudoers, Apt, Firewall) - + // Run the various mage targets + mg.Deps(Users, Sudoers, Apt, Firewall, Prometheus, Caddy) // Finally override the ssh config to disallow root login. // This is the point of no return and will create a zombie @@ -156,6 +156,25 @@ func Firewall() error { return sh.Run("ufw", "enable") } +func Prometheus() error { + log.Println("Running target Prometheus") + err := config("files/prometheus.yml", "/etc/prometheus/prometheus.yml") + if err != nil { + return err + } + return sh.Run("systemctl", "restart", "prometheus") +} + +// Copy Caddyfile to system and restart daemon +func Caddy() error { + log.Println("Running target Caddy") + err := config("files/Caddyfile", "/etc/caddy/Caddyfile") + if err != nil { + return err + } + return sh.Run("systemctl", "restart", "caddy") +} + // config() takes source template and destination file arguments and executes // the template replacing `hostname` with the system's hostname. func config(rfile string, wfile string) error { -- 2.37.1
Signed-off-by: Amolith <amolith@secluded.site> --- setup.go | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/setup.go b/setup.go index e9f4cbc..ca0a3b4 100644 --- a/setup.go +++ b/setup.go @@ -130,6 +130,11 @@ func Firewall() error { return err } + err = sh.Run("ufw", "default", "deny", "incoming") + if err != nil { + return err + } + // Allow outgoing for ntp err = sh.Run("ufw", "allow", "out", "ntp") if err != nil { @@ -147,6 +152,18 @@ func Firewall() error { return err } + // Allow incoming Sprig/relay connections + err = sh.Run("ufw", "allow", "in", "7117") + if err != nil { + return err + } + + // Allow relay to talk with other relays + err = sh.Run("ufw", "allow", "out", "7117") + if err != nil { + return err + } + // Work around a silly Linux design choice: // If this isn't set, enabling the firewall hangs all network connections, // even if they shouldn't be blocked by the firewall. -- 2.37.1
Daniel Wilkins <tekk@linuxmail.org>Do we want to allow ssh in too?
Signed-off-by: Amolith <amolith@secluded.site> --- setup.go | 34 +++++++++++++++++++++++++++++----- 1 file changed, 29 insertions(+), 5 deletions(-) diff --git a/setup.go b/setup.go index ca0a3b4..f99f401 100644 --- a/setup.go +++ b/setup.go @@ -61,14 +61,13 @@ func Users() error { } func Sudoers() error { - fmt.Println("Running target sudoers") - return sh.Run("cp", "files/sudoers", "/etc/sudoers") + log.Println("Running target sudoers") + return copy("files/sudoers", "/etc/sudoers") } func SshConfig() error { - fmt.Println("Running target ssh_config") - err := sh.Run("cp", "files/sshd_config", "/etc/ssh/sshd_config") - + log.Println("Running target ssh_config") + err := copy("files/sshd_config", "/etc/sshd/sshd_config") if err != nil { return err } @@ -192,6 +191,31 @@ func Caddy() error { return sh.Run("systemctl", "restart", "caddy") } +// copy() copies the source rfile from the embedded filesystem to the +// destination wfile on the host filesystem. +func copy(rfile string, wfile string) error { + log.Println("Copying", rfile, "to", wfile) + + // Open rfile (read file) and stores it in sfile (source file) + sfile, err := files.Open(rfile) + if err != nil { + return err + } + + // Open wfile (write file) stores it in dfile (destination file) + dfile, err := os.OpenFile(wfile, os.O_CREATE|os.O_RDWR, 0660) + if err != nil { + return err + } + + // Write sfile to filesystem and discard number of bytes written + _, err = io.Copy(dfile, sfile) + if err != nil { + return err + } + return nil +} + // config() takes source template and destination file arguments and executes // the template replacing `hostname` with the system's hostname. func config(rfile string, wfile string) error { -- 2.37.1
Signed-off-by: Amolith <amolith@secluded.site> --- setup.go | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/setup.go b/setup.go index f99f401..c8445cb 100644 --- a/setup.go +++ b/setup.go @@ -24,7 +24,7 @@ var files embed.FS // Handles initial configuration of the instance func Deploy() error { // Run the various mage targets - mg.Deps(Users, Sudoers, Apt, Firewall, Prometheus, Caddy) + mg.Deps(Users, Sudoers, Apt, Firewall, Prometheus, Caddy, Arbor) // Finally override the ssh config to disallow root login. // This is the point of no return and will create a zombie @@ -75,6 +75,25 @@ func SshConfig() error { return sh.Run("systemctl", "restart", "ssh") } +func Arbor() error { + log.Println("Running target Arbor") + err := copy("files/relay", "/usr/local/bin/relay") + if err != nil { + return err + } + err = config("files/arbor-relay.service", "/etc/systemd/system/arbor-relay.service") + if err != nil { + return err + } + + err = sh.Run("systemctl", "daemon-reload") + if err != nil { + return err + } + + return sh.Run("systemctl", "enable", "--now", "arbor-relay") +} + func Apt() error { log.Println("Running target apt") apt := sh.RunCmd("env", "DEBIAN_FRONTEND=noninteractive", "apt-get", "-qq") -- 2.37.1
Daniel Wilkins <tekk@linuxmail.org>Should this include hte service name somewhere in the sh.Run invocation? I don't run systemd so I can't test off-hand.
Signed-off-by: Amolith <amolith@secluded.site> --- setup.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.go b/setup.go index c8445cb..f67f6c1 100644 --- a/setup.go +++ b/setup.go @@ -1,4 +1,4 @@ -//+build mage +//go:build mage package main -- 2.37.1
For the useradd command, shell is not usually set by default or it defaults to /bin/sh. Signed-off-by: Amolith <amolith@secluded.site> --- setup.go | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/setup.go b/setup.go index f67f6c1..b12e45c 100644 --- a/setup.go +++ b/setup.go @@ -34,13 +34,19 @@ func Deploy() error { return nil } - +// Create default system user "user" with SSH credentials from root user and +// arbor-relay user with minimal permissions func Users() error { - fmt.Println("Running target users") + log.Println("Running target users") // Cleanup the old user if it exists. This just deletes the *user*, all data will still be intact. sh.Run("userdel", "user") - err := sh.Run("useradd", "-g", "sudo", "-m", "-d", "/home/user", "user") + err := sh.Run("useradd", "-g", "sudo", "-m", "-d", "/home/user", "-s", "/bin/bash", "user") + if err != nil { + return err + } + + err = sh.Run("useradd", "-m", "-d", "/home/user", "-s", "/bin/bash", "arbor-relay") if err != nil { return err } @@ -140,8 +146,7 @@ func Apt() error { } func Firewall() error { - fmt.Println("running target firewall") - + log.Println("Running target firewall") err := sh.Run("ufw", "default", "deny", "outgoing") if err != nil { -- 2.37.1
Signed-off-by: Amolith <amolith@secluded.site> --- go.mod | 5 +++++ go.sum | 4 ++++ 2 files changed, 9 insertions(+) create mode 100644 go.mod create mode 100644 go.sum diff --git a/go.mod b/go.mod new file mode 100644 index 0000000..7ab8e85 --- /dev/null +++ b/go.mod @@ -0,0 +1,5 @@ +module arbor-infra + +go 1.16 + +require github.com/magefile/mage v1.13.0 diff --git a/go.sum b/go.sum new file mode 100644 index 0000000..f0b89dd --- /dev/null +++ b/go.sum @@ -0,0 +1,4 @@ +github.com/magefile/mage v1.11.0 h1:C/55Ywp9BpgVVclD3lRnSYCwXTYxmSppIgLeDYlNuls= +github.com/magefile/mage v1.11.0/go.mod h1:z5UZb/iS3GoOSn0JgWuiw7dxlurVYTu+/jHXqQg881A= +github.com/magefile/mage v1.13.0 h1:XtLJl8bcCM7EFoO8FyH8XK3t7G5hQAeK+i4tq+veT9M= +github.com/magefile/mage v1.13.0/go.mod h1:z5UZb/iS3GoOSn0JgWuiw7dxlurVYTu+/jHXqQg881A= -- 2.37.1