~natpen/gus

1

Re: [PATCH] correctly handle robots.txt

Details
Message ID
<20210221192134.04BD41CC9E0C@dd49836.kasserver.com>
DKIM signature
pass
Download raw message
Hello again :)

After letting my thoughts settle for a few days i decided
that geminispace.info will not use the "gus" section in
robots.txt anymore.
It will still honor * and indexer sections like before.

This allows for a straight forward implementation without
the need for hacks around the robotfileparser. 

I don't know if this change has any practical impact. 
Unfortunately robotfileparser has no method to dump the
content of the robots.txt after reading and i don't
want to bother with it anymore.

Drop me a line if i should send a patch.

kind regards
René

Re: [PATCH] correctly handle robots.txt

Details
Message ID
<YDK2kj2pJInubIr5@goldfish.localdomain>
In-Reply-To
<20210221192134.04BD41CC9E0C@dd49836.kasserver.com> (view parent)
DKIM signature
pass
Download raw message
Hi! Sounds good - I think you're right that there's likely no real
impact either way, so I'm aligned with this approach and would welcome
the patch to keep implementations similar.

Best,
Natalie
Reply to thread Export thread (mbox)