MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain; charset=UTF-8
Subject: git.sr.ht storage growth rate
From: "Drew DeVault" <sir@cmpwn.com>
To: <~sircmpwn/sr.ht-dev@lists.sr.ht>
Date: Fri, 18 Sep 2020 11:55:32 -0400
Message-Id: <C5QM8KFLQUHN.2796RCC83HBHA@homura>

Currently git.sr.ht is growing at about 50G per week, or 5% of its
current capacity, and will run out of storage within 9 weeks. Growth is
fairly linear; the rate seems to change slowly. Might be worth expanding
the metrics.sr.ht retention rate to make long-term trends more obvious.

Some initial observations:

- tenshi is currently only giving git.sr.ht half of the disk space on
  the host, we could reduce this growth to 2.5% by just adding the other
  half (extending deadline to 21 weeks)

- sakyua1 has enough disk space to reduce the growth rate to 1% per week
  (giving us 200 weeks). Problems: hg.sr.ht is also there, we wanted to
  use sakyua1 for other stuff.

- If enabled, ZFS deduplication ratio would be about 2.3 and should
  (double check this) fit within our RAM requirements if we expanded
  git.sr.ht's share of tenshi's RAM.

- I didn't check but I don't think ZFS compression would save us much;
  git already compresses its objects. Dedupe would be a lot more
  effective.

- If tenshi's disks were reprovisioned, we could probably expand its
  capacity 4x. Combined with reclaiming the half that is unused, that'd
  be an 8x improvement, reducing the growth rate to ~0.6%/week, and
  likely giving us 2-3 years of storage runway at least.

Future work with availability and storage distribution could affect
these plans but I'd rather defer that work, so let's just buy at least
enough time to extend the deadline past when we'd expect to look at that
work.

Budget shouldn't be an issue for any of these solutions. Will probably
cost <$500 in any case.

Will revisit this and come up with a detailed action plan in 3 weeks.