~sircmpwn/gmni-devel

This thread contains a patchset. You're looking at the original emails, but you may wish to use the patch review UI. Review patch
1

[PATCH kineto] Serve robots.txt disallowing all robots

mbays
Details
Message ID
<20210825100856.5422-1-mbays@sdf.org>
DKIM signature
missing
Download raw message
Patch: +6 -0
This overrides any robots.txt file in the proxied gemini capsule, on the
basis that this is intended for gemini robots (which can be expected to
follow the robots.txt companion spec) rather than web robots.

The main purpose though for disallowing web robots is to prevent them
from crawling the proxied cross-site geminispace under /x/, since web
robots won't know even to read the robots.txt files for other capsules
proxied this way.
---
 main.go | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/main.go b/main.go
index c600985..2b227a0 100644
--- a/main.go
+++ b/main.go
@@ -583,6 +583,12 @@ func main() {
			return
		}

		if r.URL.Path == "/robots.txt" {
			w.WriteHeader(http.StatusOK)
			w.Write([]byte("User-agent: *\nDisallow: /\n"))
			return
		}

		req := gemini.Request{}
		req.URL = &url.URL{}
		req.URL.Scheme = root.Scheme
-- 
2.32.0
Details
Message ID
<CDSJKCUWI4K7.15B9QW3AD5ZMC@taiga>
In-Reply-To
<20210825100856.5422-1-mbays@sdf.org> (view parent)
DKIM signature
fail
Download raw message
DKIM signature: fail
Thanks!

To git@git.sr.ht:~sircmpwn/kineto
   a8c54c1..988a00f  master -> master
Reply to thread Export thread (mbox)