crawler: ignore 405 responses for HEAD requests

crawler: ignore 405 responses for HEAD requests v1 APPLIED

Umar Getagazov: 1
 crawler: ignore 405 responses for HEAD requests

 1 files changed, 2 insertions(+), 0 deletions(-)

Export patchset (mbox)

How do I use this?

Copy & paste the following snippet into your terminal to import this patchset into git:

curl -s https://lists.sr.ht/~sircmpwn/searchhut-devel/patches/33897/mbox | git am -3

Learn more about email & git

Reply to thread View this thread in the archives

[PATCH] crawler: ignore 405 responses for HEAD requests Export this patch

Umar Getagazov <umar@handlerug.me>

2 years ago

It also skips header checks because they usually don't have headers we
need (for example, most omit the Content-Type header).

Fixes: https://todo.sr.ht/~sircmpwn/searchhut/41
---
 crawler/index.go | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/crawler/index.go b/crawler/index.go
index 3ea7949..c2e140d 100644
--- a/crawler/index.go
+++ b/crawler/index.go
@@ -194,6 +194,8 @@ func (c *Crawler) checkResponse(resp *http.Response, page scheduledPage) bool {
 	switch resp.StatusCode {
 	case http.StatusOK:
 		// no-op
+	case http.StatusMethodNotAllowed:
+		return resp.Request.Method == "HEAD"
 	case http.StatusTooManyRequests:
 		retryAfter := resp.Header.Get("Retry-After")
 		if retryAfter == "" {
-- 
2.32.1 (Apple Git-133)

Drew DeVault <sir@cmpwn.com>

2 years ago
Thanks!

To git@git.sr.ht:~sircmpwn/searchhut
   562fecf..2b6407e  master -> master