~sircmpwn/sr.ht-discuss

12 2

redis for build.sr.ht and worker

Details
Message ID
<20190919195010.GA1196932@michaels.world>
DKIM signature
pass
Download raw message
Hi,
I was reading the builds.sr.ht installation guide, and it said to configure two redis servers. However, the sample config only has one redis server configuration value. Where should I place the other, and which should go in what key?
Thanks,
-Michael.
Details
Message ID
<BX48OTID5G5Q.5I6O5K594RE4@homura>
In-Reply-To
<20190919195010.GA1196932@michaels.world> (view parent)
DKIM signature
pass
Download raw message
Each builder should its own redis on localhost, and a shared redis. The
shared redis is the only configurable one. For hacking purposes or
single-server setups, they can be the same redis instance with different
dbs selected. Yeah, it's confusing. Sorry.
Details
Message ID
<20190919200441.GA1198734@michaels.world>
In-Reply-To
<BX48OTID5G5Q.5I6O5K594RE4@homura> (view parent)
DKIM signature
pass
Download raw message
Hi,
Ok. I noticed in the example configs (for all the services combined) that there are two redis servers with /0. Would those conflict with each other? I ask because A. I know basically nothing about redis, and B. I submitted a test .build.yml file to a git repo and it queued the job, but it hasn't been picked up and I think they are using the same redis instance for shared.
-Michael.




On Thu, Sep 19, 2019 at 03:52:04PM -0400, Drew DeVault wrote:
>Each builder should its own redis on localhost, and a shared redis. The
>shared redis is the only configurable one. For hacking purposes or
>single-server setups, they can be the same redis instance with different
>dbs selected. Yeah, it's confusing. Sorry.
Details
Message ID
<BX49BDGEHXJE.ZG4411DR1ITL@homura>
In-Reply-To
<20190919200441.GA1198734@michaels.world> (view parent)
DKIM signature
pass
Download raw message
The truth is that sr.ht uses Redis in a disorganized and shitty manner
which is begging to be refactored. Basically, it's used for these
things:

1. Volatile data store for the web services, shared among all instances
   of each service, i.e. git1.sr.ht and git2.sr.ht are loadbalanced and
   need a single redis instance, but don't (but could?) share it with
   meta.sr.ht et al.
2. Distributing tasks for webhook delivery for each service. Each one
   needs their own redis service, also shared between all instances.
3. Distributing build jobs to each build worker. Must be shared between
   builds.sr.ht and each build slave.
4. Distributing incoming emails to email processors for lists.sr.ht.

All of these can be the same redis server on different databases, but a
full sourcehut installation might run out. The config options for redis
in each config file turn one of these knobs but who knows which one.
Some of them cannot be configured and always assume the server is on
localhost. Sorry, the suckage here is pretty strong.

In practice on sr.ht upstream, each web service (git.sr.ht, lists.sr.ht,
etc) runs on its own server, alongside the webhooks processors, and have
a local redis database for both of them.  builds.sr.ht has its own
redis, but exposes it over the network to each build slave, which run on
a different database. Another redis server runs on mail.sr.ht, where
lists.sr.ht-lmtp also runs, which the main lists.sr.ht box connects to
for running the incoming mail processors.
Details
Message ID
<20190919205348.GA1209969@michaels.world>
In-Reply-To
<BX49BDGEHXJE.ZG4411DR1ITL@homura> (view parent)
DKIM signature
pass
Download raw message
Hi,
On builds.2mb.codes it claims it queued two jobs from a git push, but the worker is not picking them up. All I'm getting in the systemd journal is "starting..." and "waiting for tasks..."
It can connect to the redis on the master server, and I see the connections (4 of them), but it isn't getting anything started. Any troubleshooting I can do here?
-Michael.




On Thu, Sep 19, 2019 at 04:21:31PM -0400, Drew DeVault wrote:
>The truth is that sr.ht uses Redis in a disorganized and shitty manner
>which is begging to be refactored. Basically, it's used for these
>things:
>
>1. Volatile data store for the web services, shared among all instances
>   of each service, i.e. git1.sr.ht and git2.sr.ht are loadbalanced and
>   need a single redis instance, but don't (but could?) share it with
>   meta.sr.ht et al.
>2. Distributing tasks for webhook delivery for each service. Each one
>   needs their own redis service, also shared between all instances.
>3. Distributing build jobs to each build worker. Must be shared between
>   builds.sr.ht and each build slave.
>4. Distributing incoming emails to email processors for lists.sr.ht.
>
>All of these can be the same redis server on different databases, but a
>full sourcehut installation might run out. The config options for redis
>in each config file turn one of these knobs but who knows which one.
>Some of them cannot be configured and always assume the server is on
>localhost. Sorry, the suckage here is pretty strong.
>
>In practice on sr.ht upstream, each web service (git.sr.ht, lists.sr.ht,
>etc) runs on its own server, alongside the webhooks processors, and have
>a local redis database for both of them.  builds.sr.ht has its own
>redis, but exposes it over the network to each build slave, which run on
>a different database. Another redis server runs on mail.sr.ht, where
>lists.sr.ht-lmtp also runs, which the main lists.sr.ht box connects to
>for running the incoming mail processors.
Details
Message ID
<BX4A0RV9PRJ9.2J4MZFGBX69WJ@homura>
In-Reply-To
<20190919205348.GA1209969@michaels.world> (view parent)
DKIM signature
pass
Download raw message
Make sure that the worker is reading the jobs from the same redis server
& db that they're being submitted to (e.g. /0 vs /1).
Details
Message ID
<20190919213721.GA1218214@michaels.world>
In-Reply-To
<BX4A0RV9PRJ9.2J4MZFGBX69WJ@homura> (view parent)
DKIM signature
pass
Download raw message
They're definitely using the same database number and redis instance, minus the fact that they have to connect to different ips (localhost versus another).
Is there some command I can run, python, redis-cli, something to see the list of queue tasks?
-Michael.




On Thu, Sep 19, 2019 at 04:54:42PM -0400, Drew DeVault wrote:
>Make sure that the worker is reading the jobs from the same redis server
>& db that they're being submitted to (e.g. /0 vs /1).
Details
Message ID
<BX4AXUJVJR71.2LHJI9XYZE5Y@homura>
In-Reply-To
<20190919213721.GA1218214@michaels.world> (view parent)
DKIM signature
pass
Download raw message
On Thu Sep 19, 2019 at 2:37 PM Michael Taboada wrote:
> They're definitely using the same database number and redis instance,
> minus the fact that they have to connect to different ips (localhost
> versus another).  Is there some command I can run, python, redis-cli,
> something to see the list of queue tasks?

I think celery has some kind of monitor but I've never bothered to try
and get it working.
Details
Message ID
<20190919222454.GA1224110@michaels.world>
In-Reply-To
<BX4AXUJVJR71.2LHJI9XYZE5Y@homura> (view parent)
DKIM signature
pass
Download raw message
One other thought: I might be missing something required to run along side the worker. Right now my master server has all the main services, and my worker machine has just the builds.sr.ht worker on it. They're both running arch, so master server has all of the sr.ht packages on it except worker, and the worker machine simply has worker and images. Is this right?
They also both have redis, and the worker has docker.
-Michael.




On Thu, Sep 19, 2019 at 05:37:54PM -0400, Drew DeVault wrote:
>On Thu Sep 19, 2019 at 2:37 PM Michael Taboada wrote:
>> They're definitely using the same database number and redis instance,
>> minus the fact that they have to connect to different ips (localhost
>> versus another).  Is there some command I can run, python, redis-cli,
>> something to see the list of queue tasks?
>
>I think celery has some kind of monitor but I've never bothered to try
>and get it working.
Details
Message ID
<BX4BZ8LENKQN.33EASE94O7UGJ@homura>
In-Reply-To
<20190919222454.GA1224110@michaels.world> (view parent)
DKIM signature
pass
Download raw message
Yes, that should be sufficient. Make sure you've built at least one
image and you've set up the docker qemu container as well.
Details
Message ID
<20190919234545.GA1267280@michaels.world>
In-Reply-To
<BX4BZ8LENKQN.33EASE94O7UGJ@homura> (view parent)
DKIM signature
pass
Download raw message
Actually, that's probably the problem
Now, however, when I run the docker build command, I get:

Step 18/21 : COPY --from=0 /lib/x86_64-linux-gnu/libnss_dns-2.28.so /lib/x86_64-linux-gnu/
COPY failed: stat /var/lib/docker/overlay2/20fdde95899bc437fdd6e1b4927974328fcc4e05e6aca66f5ae4d07e78d82503/merged/lib/x86_64-linux-gnu/libnss_dns-2.28.so: no such file or directory




On Thu, Sep 19, 2019 at 06:26:44PM -0400, Drew DeVault wrote:
>Yes, that should be sufficient. Make sure you've built at least one
>image and you've set up the docker qemu container as well.
Details
Message ID
<20190919235948.GA1270196@michaels.world>
In-Reply-To
<BX4BZ8LENKQN.33EASE94O7UGJ@homura> (view parent)
DKIM signature
pass
Download raw message
Looks like libnss_dns is now at 2.29, so that's the error.
-Michael.




On Thu, Sep 19, 2019 at 06:26:44PM -0400, Drew DeVault wrote:
>Yes, that should be sufficient. Make sure you've built at least one
>image and you've set up the docker qemu container as well.
Details
Message ID
<20190920141045.GA1457126@michaels.world>
In-Reply-To
<BX4BZ8LENKQN.33EASE94O7UGJ@homura> (view parent)
DKIM signature
pass
Download raw message
Hiya,
I actually hadn't built the image of the docker image, but I did that now. I built an archlinux image and the qemu docker image, but it still won't run any queued tasks. I've tried debugging commands like celery -A buildsrht.runner status, and lots of others, and basically every one of them respondes "Error: No nodes replied within time constraint." I'm not quite sure what this means, maybe that the worker isn't actually looking for events, or maybe the build.sr.ht master server isn't  queuing them. But something's certainly off.
-Michael.





On Thu, Sep 19, 2019 at 06:26:44PM -0400, Drew DeVault wrote:
>Yes, that should be sufficient. Make sure you've built at least one
>image and you've set up the docker qemu container as well.