~sircmpwn/sr.ht-dev

This thread contains a patchset. You're looking at the original emails, but you may wish to use the patch review UI. Review patch
3 3

[PATCH builds.sr.ht 1/2] Record log cache statistics

Details
Message ID
<20210402110613.180405-1-me@ignaskiela.eu>
DKIM signature
pass
Download raw message
Patch: +12 -0
---
I think we should bump down the cache period to 1-2 days, but let's collect some
data on it first.

 buildsrht/blueprints/jobs.py | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/buildsrht/blueprints/jobs.py b/buildsrht/blueprints/jobs.py
index b0e98d9..6f092ff 100644
--- a/buildsrht/blueprints/jobs.py
@@ -13,6 +13,7 @@ from buildsrht.manifest import Manifest
from buildsrht.rss import generate_feed
from buildsrht.runner import queue_build
from buildsrht.search import apply_search
from prometheus_client import Counter
from jinja2 import Markup, escape
import sqlalchemy as sa
import hashlib
@@ -22,6 +23,15 @@ import json

jobs = Blueprint("jobs", __name__)

metrics = type("metrics", tuple(), {
    c.describe()[0].name: c
    for c in [
        Counter("buildsrht_log_fetches", "Total number of logs fetched"),
        Counter("buildsrht_log_cached", "Number of logs fetched from cache"),
    ]
})


def tags(tags):
    if not tags:
        return list()
@@ -361,9 +371,11 @@ def job_by_id(username, job_id):
        JobStatus.cancelled,
    ]
    def get_log(log_url, name, status):
        metrics.buildsrht_log_fetches.inc()
        cachekey = f"builds.sr.ht:logs:{log_url}"
        log = get_cache(cachekey)
        if log:
            metrics.buildsrht_log_cached.inc()
            log = json.loads(log)
            log["log"] = Markup(log["log"])
        if not log:
-- 
2.25.1

[PATCH builds.sr.ht 2/2] Record queued builds

Details
Message ID
<20210402110613.180405-2-me@ignaskiela.eu>
In-Reply-To
<20210402110613.180405-1-me@ignaskiela.eu> (view parent)
DKIM signature
pass
Download raw message
Patch: +4 -0
---
This will not be reliable until prometheus multiprocess mode will be enabled,
see the patch I've sent to sr.ht-packages

 buildsrht/runner.py | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/buildsrht/runner.py b/buildsrht/runner.py
index 7d5ebc6..b5e1665 100644
--- a/buildsrht/runner.py
@@ -3,6 +3,7 @@ from celery import Celery
from srht.config import cfg
from srht.database import db
from srht.email import send_email
from prometheus_client import Counter

runner = Celery('builds', broker=cfg("builds.sr.ht", "redis"), config_source={
    "CELERY_TASK_SERIALIZER": "json",
@@ -12,6 +13,8 @@ runner = Celery('builds', broker=cfg("builds.sr.ht", "redis"), config_source={
    "CELERY_TASK_PROTOCOL": 1
})

builds_queued = Counter("buildsrht_builds_queued", "Number of builds queued")

def queue_build(job, manifest):
    from buildsrht.types import JobStatus
    job.status = JobStatus.queued
@@ -23,6 +26,7 @@ def queue_build(job, manifest):
                cfg("sr.ht", "owner-email"),
                "Cryptocurrency mining attempt on builds.sr.ht")
    else:
        builds_queued.inc()
        run_build.delay(job.id, manifest.to_dict())

@runner.task
-- 
2.25.1

[builds.sr.ht/patches] build success

builds.sr.ht
Details
Message ID
<CAD6Y0OBQGHH.1K50CWN32T5YL@cirno>
In-Reply-To
<20210402110613.180405-2-me@ignaskiela.eu> (view parent)
DKIM signature
missing
Download raw message
builds.sr.ht/patches: SUCCESS in 2m49s

[Record log cache statistics][0] from [Ignas Kiela][1]

[0]: https://lists.sr.ht/~sircmpwn/sr.ht-dev/patches/21738
[1]: mailto:me@ignaskiela.eu

✓ #475369 SUCCESS builds.sr.ht/patches/archlinux.yml https://builds.sr.ht/~sircmpwn/job/475369
✓ #475370 SUCCESS builds.sr.ht/patches/debian.yml    https://builds.sr.ht/~sircmpwn/job/475370
✓ #475368 SUCCESS builds.sr.ht/patches/alpine.yml    https://builds.sr.ht/~sircmpwn/job/475368
Details
Message ID
<CAIC1ZIK1DFW.1CKIBJJA2XTN@taiga>
In-Reply-To
<20210402110613.180405-1-me@ignaskiela.eu> (view parent)
DKIM signature
fail
Download raw message
DKIM signature: fail
On Fri Apr 2, 2021 at 7:06 AM EDT, Ignas Kiela wrote:
> +metrics = type("metrics", tuple(), {
> + c.describe()[0].name: c
> + for c in [
> + Counter("buildsrht_log_fetches", "Total number of logs fetched"),
> + Counter("buildsrht_log_cached", "Number of logs fetched from cache"),

This would be better named buildsrht_logcache_hit and _miss.
Reply to thread Export thread (mbox)