Hiya, I'm having an issue with our sourcehut installation. The meta.sr.ht
service is throwing the following error:
[CRITICAL] WORKER TIMEOUT
# Versions and stuff
Install type: latest Alpine Linux (3.12.1) apk package-based install.
Versions are the latest available:
rosewater:/var/log# cat /etc/alpine-release
3.12.1
rosewater:/var/log# uname -a
Linux rosewater.cyberia.club 5.4.72-0-virt #1-Alpine SMP Mon, 19 Oct 2020 06:22:29 UTC x86_64 Linux
todo.sr.ht-0.62.8-r0 x86_64 {todo.sr.ht} (AGPLv3) [installed]
paste.sr.ht-openrc-0.11.3-r0 x86_64 {paste.sr.ht} (AGPLv3) [installed]
meta.sr.ht-0.52.7-r0 x86_64 {meta.sr.ht} (AGPLv3) [installed]
lists.sr.ht-0.47.0-r0 x86_64 {lists.sr.ht} (AGPLv3) [installed]
todo.sr.ht-openrc-0.62.8-r0 x86_64 {todo.sr.ht} (AGPLv3) [installed]
git.sr.ht-openrc-0.62.6-r0 x86_64 {git.sr.ht} (AGPLv3) [installed]
hub.sr.ht-openrc-0.11.11-r0 x86_64 {hub.sr.ht} (AGPLv3) [installed]
man.sr.ht-0.15.6-r0 x86_64 {man.sr.ht} (AGPLv3) [installed]
paste.sr.ht-0.11.3-r0 x86_64 {paste.sr.ht} (AGPLv3) [installed]
git.sr.ht-0.62.6-r0 x86_64 {git.sr.ht} (AGPLv3) [installed]
hub.sr.ht-0.11.11-r0 x86_64 {hub.sr.ht} (AGPLv3) [installed]
man.sr.ht-openrc-0.15.6-r0 x86_64 {man.sr.ht} (AGPLv3) [installed]
lists.sr.ht-openrc-0.47.0-r0 x86_64 {lists.sr.ht} (AGPLv3) [installed]
meta.sr.ht-openrc-0.52.7-r0 x86_64 {meta.sr.ht} (AGPLv3) [installed]
builds.sr.ht-0.63.10-r0 x86_64 {builds.sr.ht} (AGPLv3) [installed]
builds.sr.ht-openrc-0.63.10-r0 x86_64 {builds.sr.ht} (AGPLv3) [installed]
# Symptoms
Initially, all services started throwing 500's and logging the following:
[2020-11-23 22:59:33 +0000] [2558] [CRITICAL] WORKER TIMEOUT (pid:2933)
I figured it had to be related to meta.sr.ht::api, and I was correct. This following config was mandatory:
[meta.sr.ht::api]
internal-ipnet=127.0.0.0/8,::1/128,192.168.0.0/16,10.0.0.0/8,69.61.2.170/30
After that was implemented, every service started working per usual - except for meta.sr.ht. It is still
in the WORKER TIMEOUT state, and I cannot figure out why.
Any ideas off the top of your head? Otherwise I'm going to buckle in and do some debugging.
> Do you have the API service running? metasrht-api?
Yep, it's running and responding to queries
rosewater:/var/log# service meta.sr.ht-api status
* status: started
rosewater:/var/log# tail /var/log/meta.sr.ht-api.log
2020/11/23 23:32:35 Running on :5100
2020/11/23 23:32:35 Prometheus listening on :36949
rosewater:/var/log# netstat -plunt | grep 5100
tcp 0 0 0.0.0.0:5100 0.0.0.0:* LISTEN 4506/metasrht-api
rosewater:/var/log# curl localhost:5100/query
{"message":"Authentication error: Authorization header is required. Expected 'Authorization: Bearer \u003ctoken\u003e'"}
The webUI also seems to work fine.
However, curling the meta.sr.ht service results in a 504:
rosewater:/var/log# curl localhost:5000
<hangs indefinitely>
Hm, I'm not sure. Someone else encountered this, too, but I'm not sure
that they made any progress. I can investigate further tomorrow if you
don't mind me popping into your server to look around -
https://meta.sr.ht/~sircmpwn.keys
> Hm, I'm not sure. Someone else encountered this, too, but I'm not sure> that they made any progress. I can investigate further tomorrow if you> don't mind me popping into your server to look around -> > https://meta.sr.ht/~sircmpwn.keys
Sure, that'd be great!
ssh sircmpwn@rosewater.cyberia.club
Hello,
I had also encountered this issue. What solved it for me was two things.
One is that I had to add my meta domain to my host file and point it to
127.0.0.1 . I believe that is what solved the meta service always
crashing.
The second issue I had was that the oauth2 page would error out. For this
I had to add a section in my apache config to also proxy requests to
/query to 127.0.0.1:5100/query.
After that everything seems to work alright as far as I can see.
Marc
> The second issue I had was that the oauth2 page would error out. For this > I had to add a section in my apache config to also proxy requests to > /query to 127.0.0.1:5100/query.
This is the thing that fixed everything for me! I didn't have to mess with /etc/hosts
at all.
Using the config here as a reference:
https://git.sr.ht/~sircmpwn/sr.ht-nginx/tree/master/meta.sr.ht.conf
Other symptoms were:
- services would timeout if you tried to visit them on the web, however if you
restarted nginx occasionally some or all of the sr.ht services
would start working, except meta.sr.ht - which never worked.
- nginx threw the following 404:
"GET /query/api-meta.json HTTP/1.1" 404 4776
Thanks a ton for the help! I revoked sircmpwn's key on our system
so that he can't haxx us :-) jk
Have a good one!
j3s