I stumbled across something while reading the gddo-server code: https://git.sr.ht/~sircmpwn/gddo/tree/c4471710cd7c451d14cb969946908fb44b63b65b/item/gddo-server/main.go#L307 Any idea what this is meant to achieve?
I think it is intended to make indexing easier for Google.
I'm guessing it's a poor substitute for <a rel="nofollow">, which is standard practice for websites to signal to crawlers that a link is user-generated content and not endorsed by the website. Prevents SEO spamming.
On the whole, the robot detection thing is pretty bad. I have it disabled in production. It would be better to use things like rel="noopener nofollow" on all user-generated links.
On Mon Jan 25, 2021 at 2:02 PM EST, Drew DeVault wrote: > On the whole, the robot detection thing is pretty bad. I have it > disabled in production. It would be better to use things like > rel="noopener nofollow" on all user-generated links. It seems like the server does not increment the popularity counter of a package for robot requests. Would removing the robot detection cause a problem?
Ah, that makes sense. I think we might want to remove the popularity from the UI and just use it to silently rank results in the background. We could also use the number of packages which depend on a package as a measure of its popularity, rather than hits on the web.