cfetch/1.0
38.113.234.180 crawl1.cosmixcorp.com
voyager/1.0
38.113.234.180 crawl1.cosmixcorp.com
They have started changing useragents lately. The site says its using the Voyager useragent.
What is your crawler's HTTP user-agent string?
voyager/1.0
Thats really strange since it keeps using cfetch/1.0 most of the time.
I had been banning them by ip but will try the robots file again.
Add this to robots.txt
User-agent: voyager
Disallow: /
2 comments:
Beware of anything coming from Performance Systems Internation in Washington DC. I have blocked all of ^38. I have not found anything good coming out of the 38. range in years!
Added to my page on bad IP addresses - I was just attacked (tons of page-not-found errors) by this bot.
http://www.lifeisaprayer.com/web-design/2010/bad-annoying-ip-addresses
Post a Comment