~sircmpwn/sr.ht-discuss

5 3

Default protection for pages regarding "AI" companies?

Details
Message ID
<1e8615b6-756a-beaa-2f2b-64d70154a3c3@protonmail.com>
DKIM signature
missing
Download raw message
I saw on the Fediverse that, following the news of OpenAI providing an 
opt-out tool for robots.txt (not amused), some people have listed a 
bunch of IP ranges associated to their crawlers.

https://mathstodon.xyz/@filipw/110852364162684071

I was wondering whether it would be reasonable for sourcehut to block 
these IP ranges (and potentially those of Google, MS, etc, if they can 
be found).

Best,
Tanguy
Details
Message ID
<CUN196JXT8CX.2ACDKVWJMFC9N@taiga>
In-Reply-To
<1e8615b6-756a-beaa-2f2b-64d70154a3c3@protonmail.com> (view parent)
DKIM signature
missing
Download raw message
I went ahead and added this to robots.txt, but I'm not making any
special priority to roll out the change so it'll show up over the next
few weeks.

Not going to block any IP ranges, odds are it would result in collateral
damage.
Details
Message ID
<3f0112fa-8cc3-d691-701e-626e1c44b0ba@protonmail.com>
In-Reply-To
<CUN196JXT8CX.2ACDKVWJMFC9N@taiga> (view parent)
DKIM signature
missing
Download raw message
> I went ahead and added this to robots.txt, but I'm not making any
> special priority to roll out the change so it'll show up over the next
> few weeks.
> 
> Not going to block any IP ranges, odds are it would result in collateral
> damage.

I understand, thanks!
Details
Message ID
<169174761531.6.8777303339332517419.164251224@ploum.eu>
In-Reply-To
<CUN196JXT8CX.2ACDKVWJMFC9N@taiga> (view parent)
DKIM signature
missing
Download raw message

------- Original Message -------
On Friday, August 11th, 2023 at 11:50, Drew DeVault <sir@cmpwn.com> wrote:


> 
> 
> I went ahead and added this to robots.txt, but I'm not making any
> special priority to roll out the change so it'll show up over the next
> few weeks.

Thanks for that!

Does this means that every website hosted on sourcehut is already covered?

Or should I add the robots.txt to my build process?
Details
Message ID
<CUPMOMVNZ06F.3ODU8D16A3FZZ@taiga>
In-Reply-To
<169174761531.6.8777303339332517419.164251224@ploum.eu> (view parent)
DKIM signature
missing
Download raw message
robots.txt for pages.sr.ht is your own concern, you should add one if
you need it.
Details
Message ID
<169175640818.7.1982129982501072051.164308982@ploum.eu>
In-Reply-To
<CUPMOMVNZ06F.3ODU8D16A3FZZ@taiga> (view parent)
DKIM signature
missing
Download raw message
On 23/08/11 12:03, Drew DeVault wrote:
>robots.txt for pages.sr.ht is your own concern, you should add one if
>you need it.

Thanks for the clarification. It makes sense.
Reply to thread Export thread (mbox)