~sircmpwn/sr.ht-dev

builds.sr.ht: Simplify NetBSD image creation and remove EOL NetBSD version v5 SUPERSEDED

Michael Forney: 1
 Simplify NetBSD image creation and remove EOL NetBSD version

 8 files changed, 128 insertions(+), 110 deletions(-)
#519687 alpine.yml success
#519688 archlinux.yml success
#519689 debian.yml success
Export patchset (mbox)
How do I use this?

Copy & paste the following snippet into your terminal to import this patchset into git:

curl -s https://lists.sr.ht/~sircmpwn/sr.ht-dev/patches/23160/mbox | git am -3
Learn more about email & git

[PATCH builds.sr.ht v5] Simplify NetBSD image creation and remove EOL NetBSD version Export this patch

From: Thomas Merkel <tm@core.io>

Provide a more simple NetBSD installation method by extracting the
sets.

Additional to that:

- Remove py-anita dependency
- Remove NetBSD 7.x because of EOL
- Remove NetBSD current since it still needs more work
- Switch to CDN when possible to speedup download of resources
- Add NetBSD 9.2 resource information and script

Instead of creating a smaller filesystem image and resizing, just
create one with the final desired size. qemu-img creates a sparse
file, so no space is allocated until it is actually used.

Co-authored-by: Michael Forney <mforney@mforney.org>
---
Hi Drew,

Here's a version of genimg which I believe addresses all the problems
with the previous attempts. I have tested self-building the image
using a local builds.sr.ht instance using the following manifest:

  image: netbsd/9.x
  shell: true
  packages:
  - qemu
  sources:
  - https://git.sr.ht/~mcf/builds.sr.ht#netbsd
  environment:
    arch: amd64
    release: "9.x"
  tasks:
  - genimg: |
      cd builds.sr.ht/images/netbsd/$release
      sudo ./genimg

I have uploaded a bootstrapped netbsd/9.x image here:

https://cdn.netbsd.org/pub/NetBSD/misc/mcf/netbsd-9.2.img.qcow2.zst

netbsd/8.x doesn't self-bootstrap since there is no binary qemu
package for NetBSD 8.2, but it can be built just fine from a
netbsd/9.x image.

The image not settling was due to sshd being enabled via sed
substitution rather than setting sshd=YES. Perhaps the sed worked
for an older version of NetBSD without /etc/defaults/rc.conf.

The dkctl wedge device detection issue was caused by running from
an image that already has a GPT partition labeled 'NetBSD'.  This
causes the new wedge name to use the UUID as its name instead. To
fix this, I created the new root as 'NetBSD-new', and then renamed
it after identifying the wedge device.  However, using GPT caused
random hangs for me when preparing the root filesystem (often when
extracting the various archives). I am not sure how to debug this
issue, so I just used MBR instead. Thomas, do you have any ideas
about what might be causing this?

I also found that creating the image with the full size to begin
with avoids needing to resize the image altogether. qemu-img uses
ftruncate to create the image file, which does not actually allocate
the whole storage.

The only open question I have is, should pkgsrc be part of these
images? I left it in because it was there before, but it does make
the images significantly larger and it gets updated via cvs anyway
when you request installation of a pkgsrc package. Perhaps it could
just be downloaded and extracted at that time, rather than embedding
a snapshot from the time the image was created? It doesn't seem
like installing via pkgsrc would be used very much compared to
binary packages.
Other changes since v4:

- Use https for distribution URL when possible
- Remove current
- Fix cvs update command (-p writes files to stdout instead of
  updating them)
- Add a .gitconfig for the build user
- Remove swap file
- Enable ntpd
- Clean pkgsrc cache after installing base packages

 images/netbsd/7.x/genimg             |   6 -
 images/netbsd/8.x/genimg             |   7 +-
 images/netbsd/{7.x => 9.x}/functions |   0
 images/netbsd/9.x/genimg             |   7 +
 images/netbsd/current/genimg         |   5 -
 images/netbsd/functions              |   6 +-
 images/netbsd/genimg                 | 205 +++++++++++++++------------
 images/netbsd/latest                 |   2 +-
 8 files changed, 128 insertions(+), 110 deletions(-)
 delete mode 100755 images/netbsd/7.x/genimg
 rename images/netbsd/{7.x => 9.x}/functions (100%)
 create mode 100755 images/netbsd/9.x/genimg
 delete mode 100755 images/netbsd/current/genimg

diff --git a/images/netbsd/7.x/genimg b/images/netbsd/7.x/genimg
deleted file mode 100755
index 3208a4d..0000000
--- a/images/netbsd/7.x/genimg
@@ -1,6 +0,0 @@
#!/bin/sh -eu
arch="${1:-amd64}"
export netbsd_dist="ftp://ftp.netbsd.org/pub/NetBSD/NetBSD-7.2/$arch/"
export pkgsrc_dist="ftp://ftp.netbsd.org/pub/pkgsrc/current/pkgsrc.tar.gz"
export pkgin_dist="ftp://ftp.netbsd.org/pub/pkgsrc/packages/netbsd/$arch/7.2/All"
exec ../genimg "$@"
diff --git a/images/netbsd/8.x/genimg b/images/netbsd/8.x/genimg
index ebddf0a..f723c0f 100755
--- a/images/netbsd/8.x/genimg
+++ b/images/netbsd/8.x/genimg
@@ -1,6 +1,7 @@
#!/bin/sh -eu
arch="${1:-amd64}"
export netbsd_dist="ftp://ftp.netbsd.org/pub/NetBSD/NetBSD-8.0/$arch/"
export pkgsrc_dist="ftp://ftp.netbsd.org/pub/pkgsrc/current/pkgsrc.tar.gz"
export pkgin_dist="ftp://ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/$arch/8.0/All"
export netbsd_dist="https://cdn.netbsd.org/pub/NetBSD/NetBSD-8.2/$arch"
export netbsd_suffix="tgz"
export pkgin_dist="http://cdn.netbsd.org/pub/pkgsrc/packages/NetBSD/$arch/8.2/All"
export pkgsrc_dist="https://cdn.netbsd.org/pub/pkgsrc/current/pkgsrc.tar.gz"
exec ../genimg "$@"
diff --git a/images/netbsd/7.x/functions b/images/netbsd/9.x/functions
similarity index 100%
rename from images/netbsd/7.x/functions
rename to images/netbsd/9.x/functions
diff --git a/images/netbsd/9.x/genimg b/images/netbsd/9.x/genimg
new file mode 100755
index 0000000..95f5169
--- /dev/null
+++ b/images/netbsd/9.x/genimg
@@ -0,0 +1,7 @@
#!/bin/sh -eu
arch="${1:-amd64}"
export netbsd_dist="https://cdn.netbsd.org/pub/NetBSD/NetBSD-9.2/$arch"
export netbsd_suffix="tar.xz"
export pkgin_dist="https://cdn.netbsd.org/pub/pkgsrc/packages/NetBSD/$arch/9.2/All"
export pkgsrc_dist="https://cdn.netbsd.org/pub/pkgsrc/current/pkgsrc.tar.gz"
exec ../genimg "$@"
diff --git a/images/netbsd/current/genimg b/images/netbsd/current/genimg
deleted file mode 100755
index bb98361..0000000
--- a/images/netbsd/current/genimg
@@ -1,5 +0,0 @@
#!/bin/sh -eu
arch="${1:-amd64}"
export netbsd_dist="ftp://ftp.netbsd.org/pub/NetBSD/NetBSD-current/$arch/"
export pkgsrc_dist="ftp://ftp.netbsd.org/pub/pkgsrc/current/pkgsrc.tar.gz"
exec ../genimg "$@"
diff --git a/images/netbsd/functions b/images/netbsd/functions
index 4742273..7639aed 100644
--- a/images/netbsd/functions
+++ b/images/netbsd/functions
@@ -27,10 +27,10 @@ pkgsrc_install() {
	shift 1
	echo "Updating pkgsrc (this could take a while)..."
	guest_ssh -p $port build@localhost \
		"cd /usr/pkgsrc/ && sudo cvs update -dPp" >/dev/null 2>&1
		"cd /usr/pkgsrc/ && sudo cvs update -dP" >/dev/null 2>&1
	echo "Tip: want to speed up pkgsrc builds?" \
		"Consider installing dependencies via pkgin"
	for pkg in $@
	for pkg
	do
		# We use chronic to keep the noise to a minimum, because dear god, it
		# is noisy
@@ -50,7 +50,7 @@ install() {
	shift 1
	pkgin_pkgs=""
	pkgsrc_pkgs=""
	for pkg in $@
	for pkg
	do
		if [ -n "${pkg##*/*}" ]
		then
diff --git a/images/netbsd/genimg b/images/netbsd/genimg
index 8792e4d..3c1d5ec 100755
--- a/images/netbsd/genimg
+++ b/images/netbsd/genimg
@@ -1,12 +1,14 @@
#!/bin/sh -eux
echo "$netbsd_dist" >/dev/null # fail on -u if release unset
echo "$pkgsrc_dist" >/dev/null # fail on -u if release unset
: "$netbsd_dist"
: "$netbsd_suffix"
: "$pkgin_dist"

arch="${1:-amd64}"
sets="base comp etc kern-GENERIC man xbase"

# Functions
cleanup() {
	sync || true
	umount -R /mnt/dev/pts || true
	umount -R /mnt/dev || true
	umount -R /mnt/kern || true
	umount -R /mnt/proc || true
	umount -R /mnt/tmp || true
@@ -14,120 +16,139 @@ cleanup() {
	vndconfig -u vnd0 || true
}

mkdir -p workdir
run_root() {
	chroot /mnt /usr/bin/env \
		PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/pkg/bin:/usr/pkg/sbin \
		/bin/sh -c "$*"
}

trap cleanup EXIT

mkdir -p "$arch"
# This is really noisy/breaks terminals, hence redirecting to /dev/null
# TODO: consider just downloading the sets & configuring the bootloader
# manually
anita \
	--workdir=workdir \
	--sets=kern-GENERIC,base,etc,comp,xbase,xcomp \
	--disk-size=8G \
	--persist \
	--structured-log \
	install "$netbsd_dist" >/dev/null
mkdir -p workdir "$arch"

# Create raw image in workdir
qemu-img create -f raw workdir/wd0.img 16G

# Virtual Disk
vndconfig -c vnd0 workdir/wd0.img

mount /dev/vnd0a /mnt
mount -t null /dev /mnt/dev
mount -t null /dev/pts /mnt/dev/pts
# MBR partition creation
fdisk -fiau0 -s 169/63 vnd0
printf 'a\nunused\n\n0\nW\ny\nQ\n' | disklabel -i vnd0
mbrlabel -wr vnd0
root_disk=vnd0a

# Create file system
newfs -O 2 "$root_disk"

# Prepare and mount destination
mount "/dev/$root_disk" /mnt
mkdir /mnt/kern /mnt/proc /mnt/tmp /mnt/var

# Download and extract sets
for set in $sets
do
	if ! [ -e "workdir/$set.$netbsd_suffix" ]
	then
		ftp -o "workdir/$set.$netbsd_suffix" "$netbsd_dist/binary/sets/$set.$netbsd_suffix"
	fi
	tar -C /mnt --chroot -xpf "workdir/$set.$netbsd_suffix"
done

# Download and extract pkgsrc
if ! [ -e workdir/pkgsrc.tar.gz ]
then
	ftp -o workdir/pkgsrc.tar.gz "$pkgsrc_dist"
fi
tar -C /mnt/usr -xzf workdir/pkgsrc.tar.gz
run_root chown -R root:wsrc /usr/pkgsrc
run_root chmod -R g+w /usr/pkgsrc
cat <<EOF >/mnt/etc/mk.conf
PKG_DBDIR=	/var/db/pkg
EOF

# MAKEDEV
( cd /mnt/dev && sh MAKEDEV all )
mount -t null /kern /mnt/kern
mount -t null /proc /mnt/proc
mount -t null /tmp /mnt/tmp

run_root() {
	chroot /mnt /usr/bin/env \
		PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/pkg/bin:/usr/pkg/sbin \
		/bin/sh -c "$*"
}

sed -e 's/timeout=5/timeout=0/g' -i /mnt/boot.cfg
# fstab
cat <<EOF >/mnt/etc/fstab
/dev/ld0a	/		ffs	rw		1 1
kernfs		/kern		kernfs	rw
ptyfs		/dev/pts	ptyfs	rw
procfs		/proc		procfs	rw
tmpfs		/var/shm	tmpfs	rw,-m1777,-sram%25
EOF

run_root useradd -mG wheel build
sed -e 's;build.*;build::1000:100::0:0:,,,:/home/build:/bin/sh;g' \
	-i /mnt/etc/master.passwd
# Install and config loader
cp /mnt/usr/mdec/boot /mnt/boot
installboot -o timeout=0 "/dev/$root_disk" /mnt/usr/mdec/bootxx_ffsv2
sed -e 's/^timeout=.*/timeout=0/' -i /mnt/boot.cfg

# Create build user
run_root useradd -G wheel,wsrc build
run_root chpass -a 'build::1000:100::0:0::/home/build:/bin/ksh'
mkdir -p /mnt/home/build
cat <<EOF >/mnt/home/build/.gitconfig
[user]
name = builds.sr.ht
email = builds@sr.ht
EOF
run_root chown -R build:users /home/build

echo "nameserver 8.8.8.8" >/mnt/etc/resolv.conf
echo "nameserver 8.8.4.4" >>/mnt/etc/resolv.conf
# Network and boot config
cat <<EOF >/mnt/etc/resolv.conf
nameserver 8.8.8.8
nameserver 8.8.4.4
EOF
cat <<EOF >/mnt/etc/ifconfig.vioif0
up
10.0.2.15 netmask 255.255.255.0
EOF
echo 

# Modify ssh configuration
sed -e 's/#PermitEmptyPasswords no/PermitEmptyPasswords yes/' \
    -e 's/UsePam yes/UsePam no/' \
	-i /mnt/etc/ssh/sshd_config

# Modify rc.conf
cat <<EOF >>/mnt/etc/rc.conf
hostname=build
defaultroute="10.0.2.2"
resize_root=YES
no_swap=YES
postfix=NO
sshd=YES
ntpd=YES
EOF

cat <<EOF >/mnt/etc/fstab
/dev/ld0a		/	ffs	rw		 1 1
/dev/ld0b		none	swap	sw,dp		 0 0
kernfs		/kern	kernfs	rw
ptyfs		/dev/pts	ptyfs	rw
procfs		/proc	procfs	rw
/dev/cd0a		/cdrom	cd9660	ro,noauto
tmpfs		/var/shm	tmpfs	rw,-m1777,-sram%25
sed -e 's/^rc_configured=NO/rc_configured=YES/' \
	-i /mnt/etc/rc.conf

run_root pkg_add "$pkgin_dist/pkgin"
echo "$pkgin_dist" > /mnt/usr/pkg/etc/pkgin/repositories.conf
run_root pkgin -y update
# TODO: Remove bash
run_root pkgin -y install sudo gnupg git-base mercurial \
	moreutils mozilla-rootcerts bash
run_root pkgin clean
run_root mozilla-rootcerts install

cat <<EOF >/mnt/etc/login.conf
default:\
	:path=/bin /sbin /usr/bin /usr/sbin /usr/pkg/bin /usr/pkg/sbin /usr/local/bin /usr/local/sbin:
EOF

sed -e 's/#PermitEmptyPasswords no/PermitEmptyPasswords yes/' \
	-i /mnt/etc/ssh/sshd_config
sed -e 's/UsePam yes/UsePam no/' \
	-i /mnt/etc/ssh/sshd_config

sed -e 's/sshd=NO/sshd=YES/g' -i /mnt/etc/defaults/rc.conf

if ! [ -e pkgsrc.tar.gz ]
then
	ftp "$pkgsrc_dist"
fi

tar -xzf pkgsrc.tar.gz -C /mnt/usr
pkgsrc_build() {
	pkg="$1"
	run_root "cd /usr/pkgsrc/$pkg && make"
	run_root "cd /usr/pkgsrc/$pkg && make install"
	run_root "cd /usr/pkgsrc/$pkg && make clean clean-depends"
}

# pkgin is not supported for NetBSD-current
if [ -n "${pkgin_dist:-}" ]
then
	pkgsrc_build pkgtools/pkgin
	echo "$pkgin_dist" >/mnt/usr/pkg/etc/pkgin/repositories.conf
	run_root pkgin update
	# TODO: Remove bash
	run_root pkgin -y install sudo gnupg git-base mercurial \
		moreutils mozilla-rootcerts bash
else
	# TODO: Remove bash
	for pkg in security/sudo security/gnupg devel/git-base misc/moreutils \
		security/mozilla-rootcerts shells/bash devel/mercurial
	do
		pkgsrc_build "$pkg"
	done
fi
run_root chown -R build /usr/pkgsrc/
run_root /usr/pkg/sbin/mozilla-rootcerts install

cat <<"EOF" >/mnt/home/build/.profile
PATH=$HOME/bin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/X11R7/bin:/usr/pkg/bin
PATH=${PATH}:/usr/pkg/sbin:/usr/games:/usr/local/bin:/usr/local/sbin
export PATH
export ENV=$HOME/.env
cat <<EOF >/mnt/usr/pkg/etc/sudoers
root ALL=(ALL) ALL
%wheel ALL=(ALL) NOPASSWD: ALL
EOF

printf '%s\n' "%wheel ALL=(ALL) NOPASSWD: ALL" >> /mnt/usr/pkg/etc/sudoers

rm /mnt/etc/motd
touch /mnt/firstboot

cleanup
trap : EXIT
trap - EXIT

qemu-img convert -f raw -O qcow2 workdir/wd0.img "$arch"/root.img.qcow2
qemu-img resize "$arch"/root.img.qcow2 16G
qemu-img convert -f raw -O qcow2 workdir/wd0.img "$arch/root.img.qcow2"
rm workdir/wd0.img
diff --git a/images/netbsd/latest b/images/netbsd/latest
index 9aa5586..fc3ec56 120000
--- a/images/netbsd/latest
+++ b/images/netbsd/latest
@@ -1 +1 @@
8.x
\ No newline at end of file
9.x
\ No newline at end of file
-- 
2.31.1
builds.sr.ht/patches: SUCCESS in 2m48s

[Simplify NetBSD image creation and remove EOL NetBSD version][0] v5 from [Michael Forney][1]

[0]: https://lists.sr.ht/~sircmpwn/sr.ht-dev/patches/23160
[1]: mailto:mforney@mforney.org

✓ #519688 SUCCESS builds.sr.ht/patches/archlinux.yml https://builds.sr.ht/~sircmpwn/job/519688
✓ #519687 SUCCESS builds.sr.ht/patches/alpine.yml    https://builds.sr.ht/~sircmpwn/job/519687
✓ #519689 SUCCESS builds.sr.ht/patches/debian.yml    https://builds.sr.ht/~sircmpwn/job/519689
Nice work! Can you do a brief write-up of how to do the full bootstrap
from scratch? I assume it's as simple as setting up a NetBSD VM and
running the genimg script, but just want to double check to save myself
the effort of troubleshooting if it's more involved.

On Fri Jun 4, 2021 at 6:37 PM EDT, Michael Forney wrote: