Nov 25, 2006

converacrawler/0.9d bot

converacrawler/0.9d (+

This bot came in and refused to take no for a answer it tried to load every page I had. So its clear that it didn't spider my site to get those links it got them from google or somewhere else.

converacrawler was orginaly banned as a email harvestor but it now looks like a real search site at

They claim you can add this to robots.txt.

User-agent: ConveraCrawler
Disallow: /

No comments: