Hey, Do you know nlnet.nl ? There, you could ask for funding to develop for instance an in-house crawler. They support already seaerchx with funding (or have, don't know), so you could explain that many search engines block the access of searchx instances. This is at least the experience of Mike kuketz, who has created a really popular searchx instance in germany. It was so popular, that he had to regularly change the IP adress to avoid being blocked. I think it would be great a federated network of crawlers. So there would be many different and independent instances earch running their own crawler. And all would talk to each other and exchange the crawling-results. Kind of like yacy, just with high but with a few powerful instances. I think this is the most stable solution to the search engine problem. It is the most independent from other search engines and probably the most succesful because of the computing power behind. At least compared to yacy, which has the problem that the results are small because the crawlers are not powerful. Here is the open call for funding, deadline is 1st of june. https://nlnet.nl/news/2020/20200401-call.html
On Thu May 21, 2020 at 2:07 PM CEST, toogley wrote: > Hey, > > Do you know nlnet.nl ? There, you could ask for funding to develop for > instance an in-house crawler. i think it would be the most sensible solution to develop that as independent from other backends/frontends. So that way, it could be used as par of searchx instances, but also as part of yacy instances but also on its own. I think we (in the free software scene) have the computing power to create an independent crawler, we are just missing the software to run. I also think it would be great if it would be possible to just focus on the crawling. So many people probably have some computing time on their servers available but don't want to run a search-engine on that. So with this idea, they could just run the crawler from time to time, and feed the results back into the network.