Exalead Snapshots your site and lists it in its search system. It also tries to hot link all your images in a page view window. Sites like mine using hotlink protection will display a image theft notice when they do this.
I thought I has stoped this snapshot bot without stoping its crawler but they have again changed the useragent for it.
See here and here for more.
block useragents.
NG/2.0,Image crawler
NG/4.0,image crawler
Robot does not comply with simple basic robots.txt commands to not load images.
User-agent: *
Disallow: /images/
Web robots are visiting sites to hack, spam, email harvest and to scrap your website contect for profit.
This blog is an atempt to keep track of them and to help webmasters by listing the abuse in google.
Hello,
ReplyDeleteSince then we had not had any bad feedback from webmasters for our preview. But I understand your point, we will add the ability for the webmasters to refuse their pages to be previewed and thumbnailed by the use of these two meta:
meta name="robots" content="nopreview"
or
meta name="robots" content="nothumbnail"
It should be online very soon.
Would that be ok for you ?
No thats not good enough you are violating robots.txt
ReplyDeleteWe have restricted bots from loading images by placing
User-agent: *
Disallow: /images/
in the robots.txt and you are violating this rule. Please comply with robots.txt file. As far as hotlinking you need to stop that for all sites because its theft of bandwidth.