~sircmpwn/sr.ht-discuss

7 3

I upgraded my sr.ht install and the meta.sr.ht process won't start properly

Details
Message ID
<C7B368CTVDCU.1PPIR61MWOG2B@nostromo>
DKIM signature
fail
Download raw message
DKIM signature: fail
Hiya, I'm having an issue with our sourcehut installation. The meta.sr.ht
service is throwing the following error:

[CRITICAL] WORKER TIMEOUT

# Versions and stuff

Install type: latest Alpine Linux (3.12.1) apk package-based install.

Versions are the latest available:

rosewater:/var/log# cat /etc/alpine-release
3.12.1
rosewater:/var/log# uname -a
Linux rosewater.cyberia.club 5.4.72-0-virt #1-Alpine SMP Mon, 19 Oct 2020 06:22:29 UTC x86_64 Linux

todo.sr.ht-0.62.8-r0 x86_64 {todo.sr.ht} (AGPLv3) [installed]
paste.sr.ht-openrc-0.11.3-r0 x86_64 {paste.sr.ht} (AGPLv3) [installed]
meta.sr.ht-0.52.7-r0 x86_64 {meta.sr.ht} (AGPLv3) [installed]
lists.sr.ht-0.47.0-r0 x86_64 {lists.sr.ht} (AGPLv3) [installed]
todo.sr.ht-openrc-0.62.8-r0 x86_64 {todo.sr.ht} (AGPLv3) [installed]
git.sr.ht-openrc-0.62.6-r0 x86_64 {git.sr.ht} (AGPLv3) [installed]
hub.sr.ht-openrc-0.11.11-r0 x86_64 {hub.sr.ht} (AGPLv3) [installed]
man.sr.ht-0.15.6-r0 x86_64 {man.sr.ht} (AGPLv3) [installed]
paste.sr.ht-0.11.3-r0 x86_64 {paste.sr.ht} (AGPLv3) [installed]
git.sr.ht-0.62.6-r0 x86_64 {git.sr.ht} (AGPLv3) [installed]
hub.sr.ht-0.11.11-r0 x86_64 {hub.sr.ht} (AGPLv3) [installed]
man.sr.ht-openrc-0.15.6-r0 x86_64 {man.sr.ht} (AGPLv3) [installed]
lists.sr.ht-openrc-0.47.0-r0 x86_64 {lists.sr.ht} (AGPLv3) [installed]
meta.sr.ht-openrc-0.52.7-r0 x86_64 {meta.sr.ht} (AGPLv3) [installed]
builds.sr.ht-0.63.10-r0 x86_64 {builds.sr.ht} (AGPLv3) [installed]
builds.sr.ht-openrc-0.63.10-r0 x86_64 {builds.sr.ht} (AGPLv3) [installed]

# Symptoms

Initially, all services started throwing 500's and logging the following:

 [2020-11-23 22:59:33 +0000] [2558] [CRITICAL] WORKER TIMEOUT (pid:2933)

I figured it had to be related to meta.sr.ht::api, and I was correct. This following config was mandatory:

[meta.sr.ht::api]
internal-ipnet=127.0.0.0/8,::1/128,192.168.0.0/16,10.0.0.0/8,69.61.2.170/30

After that was implemented, every service started working per usual - except for meta.sr.ht. It is still
in the WORKER TIMEOUT state, and I cannot figure out why.

Any ideas off the top of your head? Otherwise I'm going to buckle in and do some debugging.
Details
Message ID
<C7B3N4OJXPSS.16J7XXSKCT4UF@taiga>
In-Reply-To
<C7B368CTVDCU.1PPIR61MWOG2B@nostromo> (view parent)
DKIM signature
fail
Download raw message
DKIM signature: fail
Do you have the API service running? metasrht-api?
Details
Message ID
<C7B3NVXDIP50.3DDZIPM0JXNVS@nostromo>
In-Reply-To
<C7B3N4OJXPSS.16J7XXSKCT4UF@taiga> (view parent)
DKIM signature
fail
Download raw message
DKIM signature: fail
> Do you have the API service running? metasrht-api?

Yep, it's running and responding to queries

rosewater:/var/log# service meta.sr.ht-api status
 * status: started

rosewater:/var/log# tail /var/log/meta.sr.ht-api.log
2020/11/23 23:32:35 Running on :5100
2020/11/23 23:32:35 Prometheus listening on :36949

rosewater:/var/log# netstat -plunt | grep 5100
tcp        0      0 0.0.0.0:5100            0.0.0.0:*               LISTEN      4506/metasrht-api

rosewater:/var/log# curl localhost:5100/query
{"message":"Authentication error: Authorization header is required. Expected 'Authorization: Bearer \u003ctoken\u003e'"}

The webUI also seems to work fine.

However, curling the meta.sr.ht service results in a 504:

rosewater:/var/log# curl localhost:5000
<hangs indefinitely>
Details
Message ID
<C7B3SSL6KN01.20C1AGOI2MH9S@taiga>
In-Reply-To
<C7B3NVXDIP50.3DDZIPM0JXNVS@nostromo> (view parent)
DKIM signature
fail
Download raw message
DKIM signature: fail
Hm, I'm not sure. Someone else encountered this, too, but I'm not sure
that they made any progress. I can investigate further tomorrow if you
don't mind me popping into your server to look around -

https://meta.sr.ht/~sircmpwn.keys
Details
Message ID
<C7B3U4MDHGAD.343VJ20KHJU5T@nostromo>
In-Reply-To
<C7B3SSL6KN01.20C1AGOI2MH9S@taiga> (view parent)
DKIM signature
fail
Download raw message
DKIM signature: fail
> Hm, I'm not sure. Someone else encountered this, too, but I'm not sure
> that they made any progress. I can investigate further tomorrow if you
> don't mind me popping into your server to look around -
> 
> https://meta.sr.ht/~sircmpwn.keys

Sure, that'd be great!

ssh sircmpwn@rosewater.cyberia.club
Details
Message ID
<08b5eb46-6f4f-4ef1-b6db-b78bc2d0298a@marcg.pizza>
In-Reply-To
<C7B3U4MDHGAD.343VJ20KHJU5T@nostromo> (view parent)
DKIM signature
missing
Download raw message
Hello,

I had also encountered this issue. What solved it for me was two things. 
One is that I had to add my meta domain to my host file and point it to 
127.0.0.1 . I believe that is what solved the meta service always 
crashing.

The second issue I had was that the oauth2 page would error out. For this 
I had to add a section in my apache config to also proxy requests to 
/query to 127.0.0.1:5100/query.

After that everything seems to work alright as far as I can see.

Marc
Details
Message ID
<C7B6XZX9SOQA.2WI2LOMTEP8KC@nostromo>
In-Reply-To
<08b5eb46-6f4f-4ef1-b6db-b78bc2d0298a@marcg.pizza> (view parent)
DKIM signature
fail
Download raw message
DKIM signature: fail
> The second issue I had was that the oauth2 page would error out. For this 
> I had to add a section in my apache config to also proxy requests to 
> /query to 127.0.0.1:5100/query.

This is the thing that fixed everything for me! I didn't have to mess with /etc/hosts
at all.

Using the config here as a reference:
https://git.sr.ht/~sircmpwn/sr.ht-nginx/tree/master/meta.sr.ht.conf

Other symptoms were:

- services would timeout if you tried to visit them on the web, however if you 
  restarted nginx occasionally some or all of the sr.ht services 
  would start working, except meta.sr.ht - which never worked.
- nginx threw the following 404:

"GET /query/api-meta.json HTTP/1.1" 404 4776

Thanks a ton for the help! I revoked sircmpwn's key on our system
so that he can't haxx us :-) jk

Have a good one!

j3s
Details
Message ID
<C7BMP0H7NI6Q.IDIWA1M2ZU0C@taiga>
In-Reply-To
<C7B6XZX9SOQA.2WI2LOMTEP8KC@nostromo> (view parent)
DKIM signature
fail
Download raw message
DKIM signature: fail
Thanks for narrowing this down, I just sent an email to sr.ht-admins
with the details.
Reply to thread Export thread (mbox)