Nov 6, 2006 Is a Health search sytem



They have started changing useragents lately. The site says its using the Voyager useragent.
What is your crawler's HTTP user-agent string?


Thats really strange since it keeps using cfetch/1.0 most of the time.
I had been banning them by ip but will try the robots file again.

Add this to robots.txt
User-agent: voyager
Disallow: /


Anonymous said...

Beware of anything coming from Performance Systems Internation in Washington DC. I have blocked all of ^38. I have not found anything good coming out of the 38. range in years!

Jeff Geerling said...

Added to my page on bad IP addresses - I was just attacked (tons of page-not-found errors) by this bot.