I've been looking to revive a project that might end up using 5 or so 
pipelines (building packages for Fedora current and next release, Arch, 
Debian, and a separate pipeline to build container images that all of 
the packages use). Currently, I know that builds.sr.ht will currently 
run up to 4 build manifests at once, choosing at random if more than 4 
are present. Although I found the change where this was added, it didn't 
really have any rationale: 
https://git.sr.ht/~sircmpwn/dispatch.sr.ht/commit/2280641828bc352729ac5da0886bbb36d3b5759c#dispatchsrht/builds.py

Now, *technically* this doesn't seem to apply to the API, so I could 
have two build manifests, but one of them just queues the 4 package 
build jobs and monitors their status. I'm not sure why the limit was 
added in the first place, if it was e.g. more to stop crypto miners or 
random service abuse (I do also have a paid account). I also don't want 
to be abusing the awesome sr.ht services, although I doubt I'd have more 
than, say, a handful of builds per week (if I'm active!).

TLDR: I know the 4 build limit exists, and I know I can work around it, 
but I'm not sure if that would be considered a bad / abusive practice, 
partly because I don't know the background of the addition in the first 
place. I would appreciate any guidance on this!

-- 
Ryan
https://refi64.com/

> if it was e.g. more to stop crypto miners or 
> random service abuse (I do also have a paid account)
While i'm not positive if this was the reason, the sourcehut team was 
dealing with crypto miners in builds before.

Parallel build slots are the most expensive resource you can use on
sourcehut. We have a total of 32 parallel build slots to share with the
whole of sourcehut right now. Using a lot of them at once can quickly
lead to your project monopolozing all or most of the build slots.

Do you really get a lot of value from building 5 different packages for
5 different OSes for every single commit to your repo? What fraction of
those are actually going to be used by anyone? What bugs will they find
in one flavor of Linux but not in one slightly different flavor of
Linux?

In general, most projects with this many builds have over-provisioned
their CI above and beyond what's actually necessary for the goals of the
project.

On Sun Feb 20, 2022 at 12:26 AM CET, Ryan Gonzalez wrote:
> Now, *technically* this doesn't seem to apply to the API, so I could 
> have two build manifests, but one of them just queues the 4 package 
> build jobs and monitors their status.

Don't do this, please, circumventing anti-abuse measures is grounds for
account termination.

On Sun Feb 20, 2022 at 8:45 AM GMT, Drew DeVault wrote:
> In general, most projects with this many builds have over-provisioned
> their CI above and beyond what's actually necessary for the goals of the
> project.

I had to sacrifice Arch Linux in ~rjarry/aerc so we could build
on OpenBSD. I imagine if we ever get builds working on Plan 9 or
other systems, we might end up with a single Linux build. Which
is probably fine.

Do you know why we originally had 4 Linux builds?

One solution might be allowing us to dispatch to additional
builds.sr.ht instances. I could host one myself for aerc and my
own projects that has a much higher limit. Possibly it could
include other operating systems like Haiku.

If I were to look into patches to have SourceHut support
something like this:

    % cat .builds/haiku.yaml
    ---
    server: builds.ancarda.net
    image: haiku/beta3
    [...rest of build manifest here...]

Would that be okay? Then when you git push you get everything:

    remote: https://builds.sr.ht/... [alpine.yaml]
    remote: https://builds.sr.ht/... [openbsd.yaml]
    remote: https://builds.ancarda.net/... [haiku.yaml]

On Sun Feb 20, 2022 at 3:07 PM CET, Mark Dain wrote:
> If I were to look into patches to have SourceHut support
> something like this:
>
>     % cat .builds/haiku.yaml
>     ---
>     server: builds.ancarda.net
>     image: haiku/beta3
>     [...rest of build manifest here...]
>
> Would that be okay? Then when you git push you get everything:
>
>     remote: https://builds.sr.ht/... [alpine.yaml]
>     remote: https://builds.sr.ht/... [openbsd.yaml]
>     remote: https://builds.ancarda.net/... [haiku.yaml]

I'm not sure exactly what that would look like in terms of
implementation, but I would accept this patch if someone figures it out.

On 2/20/22 02:45, Drew DeVault wrote:
> Parallel build slots are the most expensive resource you can use on
> sourcehut. We have a total of 32 parallel build slots to share with the
> whole of sourcehut right now. Using a lot of them at once can quickly
> lead to your project monopolozing all or most of the build slots.

Ah, that does make sense. Out of curiosity, how does *length* affect 
this? Technically, I could bundle them together into 1 or 2 builds, 
using Vagrant here, but that might take a hot minute longer (not sure 
about #s yet, I'm redoing all the tests). I *thought* I had read once 
that single long builds are discouraged in favor of parallel builds, but 
I'm guessing now I misunderstood and that only really applied to *short* 
parallel builds?

Failing that, if the other poster doesn't take on trying to write a 
patch for custom builds.sr.ht instances, I might have to dive in and see 
how it goes! That would certainly be the cleanest alternative otherwise.

>
> Do you really get a lot of value from building 5 different packages for
> 5 different OSes for every single commit to your repo? What fraction of
> those are actually going to be used by anyone? What bugs will they find
> in one flavor of Linux but not in one slightly different flavor of
> Linux?
Unfortunately, the project does revolve around very particular aspects 
of the system, e.g. integrating with firewalld on Fedora vs UFW on 
Debian, SELinux compat on Fedora, it always breaks on Arch first because 
rolling, etc.
>
> In general, most projects with this many builds have over-provisioned
> their CI above and beyond what's actually necessary for the goals of the
> project.
>
> On Sun Feb 20, 2022 at 12:26 AM CET, Ryan Gonzalez wrote:
>> Now, *technically* this doesn't seem to apply to the API, so I could
>> have two build manifests, but one of them just queues the 4 package
>> build jobs and monitors their status.
> Don't do this, please, circumventing anti-abuse measures is grounds for
> account termination.
I totally understand, this is why I wanted to ask in the first place.

-- 
--
Ryan
https://refi64.com/

On Mon Feb 21, 2022 at 3:16 AM CET, Ryan Gonzalez wrote:
> Ah, that does make sense. Out of curiosity, how does *length* affect 
> this?

Build length is fine, we have a 4 hour limit and longer builds are not
an issue.

> Unfortunately, the project does revolve around very particular aspects 
> of the system, e.g. integrating with firewalld on Fedora vs UFW on 
> Debian, SELinux compat on Fedora, it always breaks on Arch first because 
> rolling, etc.

Makes sense.