~sircmpwn/sr.ht-dev

This thread contains a patchset. You're looking at the original emails, but you may wish to use the patch review UI. Review patch
6 2

[PATCH metrics.sr.ht] Add an alert for high rate of server errors

Details
Message ID
<20200628133041.952-1-me@ignaskiela.eu>
DKIM signature
fail
Download raw message
Patch: +11 -0 DKIM signature: fail
---
 service_rules.yml | 11 +++++++++++
 1 file changed, 11 insertions(+)
 create mode 100644 service_rules.yml

diff --git a/service_rules.yml b/service_rules.yml
new file mode 100644
index 0000000..26fedf0
--- /dev/null
+++ b/service_rules.yml
@@ -0,0 +1,11 @@
# vim: tw=2 sw=2 :
groups:
- name: service
  rules:
  - alert: High rate of 500 errors
    expr: rate(http_requests_total{status="500"}[10m]) > 5 / 60
    for: 2m
    labels:
      severity: important
    annotations:
      summary: "{{ $labels.instance }} has a high rate of 500 errors"
-- 
2.17.1


- Email domain proudly hosted at https://migadu.com
Details
Message ID
<C3SRVWZ0UM3A.1KDE8Z3ULVBFK@homura>
In-Reply-To
<20200628133041.952-1-me@ignaskiela.eu> (view parent)
DKIM signature
pass
Download raw message
Thanks! Can you bump this to "urgent" severity?
Details
Message ID
<C3SRWA2GL8UL.CGL3O6JONKE6@homura>
In-Reply-To
<20200628133041.952-1-me@ignaskiela.eu> (view parent)
DKIM signature
pass
Download raw message
And increase the interval to 5m as well.

[PATCH metrics.sr.ht v2] Add an alert for high rate of server errors

Details
Message ID
<20200628135551.4784-1-me@ignaskiela.eu>
In-Reply-To
<20200628133041.952-1-me@ignaskiela.eu> (view parent)
DKIM signature
fail
Download raw message
Patch: +11 -0 DKIM signature: fail
---
 service_rules.yml | 11 +++++++++++
 1 file changed, 11 insertions(+)
 create mode 100644 service_rules.yml

diff --git a/service_rules.yml b/service_rules.yml
new file mode 100644
index 0000000..2a71244
--- /dev/null
+++ b/service_rules.yml
@@ -0,0 +1,11 @@
# vim: tw=2 sw=2 :
groups:
- name: service
  rules:
  - alert: High rate of 500 errors
    expr: rate(http_requests_total{status="500"}[10m]) > 5 / 60
    for: 5m
    labels:
      severity: urgent
    annotations:
      summary: "{{ $labels.instance }} has a high rate of 500 errors"
-- 
2.17.1


- Email domain proudly hosted at https://migadu.com

Re: [PATCH metrics.sr.ht v2] Add an alert for high rate of server errors

Details
Message ID
<C3SSCYGXXARA.EJKCGBGAADU6@homura>
In-Reply-To
<20200628135551.4784-1-me@ignaskiela.eu> (view parent)
DKIM signature
pass
Download raw message
Thanks!

To git@git.sr.ht:~sircmpwn/metrics.sr.ht
   9c1389a..3435b2c  master -> master

Re: [PATCH metrics.sr.ht v2] Add an alert for high rate of server errors

Details
Message ID
<ec2fce64-38a8-40e0-d04f-96889d54b8ca@ignaskiela.eu>
In-Reply-To
<C3SSCYGXXARA.EJKCGBGAADU6@homura> (view parent)
DKIM signature
fail
Download raw message
DKIM signature: fail
Looks like this rule got lost somewhere during the deployment.

https://builds.sr.ht/~sircmpwn/job/242956##task-package-23

-- Email domain proudly hosted at https://migadu.com

Re: [PATCH metrics.sr.ht v2] Add an alert for high rate of server errors

Details
Message ID
<C3ST1SH65E8H.1GNXP73TGL4K5@homura>
In-Reply-To
<ec2fce64-38a8-40e0-d04f-96889d54b8ca@ignaskiela.eu> (view parent)
DKIM signature
pass
Download raw message
Nah, new rules just need manual intervention to enable (they have to be
added to prometheus.yml). I just took care of it.
Reply to thread Export thread (mbox)