~danskeren/ask.moe

funding to develop a independent crawler. Deadline: 1st june 2020

Details
Message ID
<K54SAQ.QYAABFCO52HC@ask.moe>
DKIM signature
missing
Download raw message
Hmm.. Migadu didn't forward this email to my mailbox :/
Better look into some alternative.. perhaps ProtonMail.

Never heard of nlnet.nl, but they look amazing.
Will definitely submit an application. Thanks!

 > many search engines block the access of searchx instances

That's true. I wasn't surprised to see Google being
quick to CAPTCHA all requests. After all, even with my
regular browser then I also have to solve a few CAPTCHAs
before every requests. But it's been quite disappointing
to see most of the search engines that claim to be privacy-
focused do the same, especially for queries that have
without a doubt been cached.

With regard to the in-house crawler. It's definitely something
I want to do as well. I will initially focus on the category-
specific search engines, because they will be much easier to
implement, they can be implemented on a low budget (at least
for some of the categories), and I think they will be more
valuable.
But I do envision a public search index, accessible to all,
that only allowed websites that didn't engage in user tracking,
SEO spam, etc.
It's not exactly the same as you described, but if there is
a demand then it could be expanded to include all websites,
including the ones I'd want to see blacklisted from Ask.Moe
for privacy-related reasons.

 > i think it would be the most sensible solution to develop that as
 > independent from other backends/frontends.

Agreed. The category-specific search indexes will be separated
as well.



-- Email domain proudly hosted at https://migadu.com
Reply to thread Export thread (mbox)