converacrawler/0.9d (+http://www.authoritativeweb.com/crawl)
63.241.61.7 7-9745.san2.attens.net
This bot came in and refused to take no for a answer it tried to load every page I had. So its clear that it didn't spider my site to get those links it got them from google or somewhere else.
converacrawler was orginaly banned as a email harvestor but it now looks like a real search site at www.govmine.com.
They claim you can add this to robots.txt.
User-agent: ConveraCrawler
Disallow: /
Subscribe to:
Post Comments (Atom)
No comments:
Post a Comment