After banning the domain amazonaws.com because they are hosting bots.
I get all of this.
Agent: webclient
75.101.206.181 ec2-75-101-206-181.compute-1.amazonaws.com
Agent: webclient
75.101.206.181 ec2-75-101-206-181.compute-1.amazonaws.com
Agent: mozilla/4.0 (compatible; msie 7.0; windows nt 5.1; .net clr 1.1.4322)
67.202.31.132 ec2-67-202-31-132.compute-1.amazonaws.com
Agent: mozilla/4.0 (compatible; msie 7.0; windows nt 5.1; .net clr 1.1.4322; .net clr 2.0.50727)
67.202.31.132 ec2-67-202-31-132.compute-1.amazonaws.com
Agent: mozilla/4.0 (compatible; msie 7.0; windows nt 5.1; .net clr 1.1.4322)
67.202.57.15 ec2-67-202-57-15.compute-1.amazonaws.com
Agent: mozilla/4.0 (compatible; msie 7.0; windows nt 5.1; .net clr 1.1.4322; .net clr 2.0.50727)
67.202.57.15 ec2-67-202-57-15.compute-1.amazonaws.com
Agent: mozilla/4.0 (compatible; msie 7.0; windows nt 5.1; .net clr 1.1.4322; .net clr 2.0.50727)
67.202.57.15 ec2-67-202-57-15.compute-1.amazonaws.com
Agent: Mozilla/5.0 (compatible; zermelo; +http://www.powerset.com) [email:paul@page-store.com-crawl@powerset.com]
72.44.49.121 ec2-72-44-49-121.z-1.compute-1.amazonaws.com
Agent: AideRSS/1.0 (aiderss.com); * subscribers
67.202.34.44 ec2-67-202-34-44.compute-1.amazonaws.com
-----Update AideRSS just does not get it that they have been blocked.
67.202.23.122 ec2-67-202-23-122.compute-1.amazonaws.com
[06-17-2008-16:07:52] Scan Blacklist Domain Ban: amazonaws.com
Agent: AideRSS/1.0 (aiderss.com); * subscribers
75.101.226.160 ec2-75-101-226-160.compute-1.amazonaws.com
[06-17-2008-16:09:04] Scan Blacklist Domain Ban: amazonaws.com
Agent: AideRSS/1.0 (aiderss.com); * subscribers
75.101.219.174 ec2-75-101-219-174.compute-1.amazonaws.com
[06-17-2008-16:09:19] Scan Blacklist Domain Ban: amazonaws.com
Agent: AideRSS/1.0 (aiderss.com); * subscribers
67.202.21.42 ec2-67-202-21-42.compute-1.amazonaws.com
[06-17-2008-16:09:22] Scan Blacklist Domain Ban: amazonaws.com
Agent: AideRSS/1.0 (aiderss.com); * subscribers
67.202.23.83 ec2-67-202-23-83.compute-1.amazonaws.com
[06-17-2008-16:09:29] Scan Blacklist Domain Ban: amazonaws.com
Agent: AideRSS/1.0 (aiderss.com); * subscribers
75.101.211.7 ec2-75-101-211-7.compute-1.amazonaws.com
[06-17-2008-16:09:35] Scan Blacklist Domain Ban: amazonaws.com
Agent: AideRSS/1.0 (aiderss.com); * subscribers
75.101.244.65 ec2-75-101-244-65.compute-1.amazonaws.com
Agent: AideRSS/1.0 (aiderss.com); * subscribers
67.202.61.94 ec2-67-202-61-94.compute-1.amazonaws.com
Update
67.202.31.132 is BLACKLISTED by dnsbl.njabl.org for spam
67.202.61.94 is BLACKLISTED by dnsbl.njabl.org for spam
67.202.23.83 is BLACKLISTED by dnsbl.njabl.org for spam
67.202.21.42 is BLACKLISTED by dnsbl.njabl.org for spam
67.202.23.122 is BLACKLISTED by dnsbl.njabl.org for spam
67.202.34.44 is BLACKLISTED by dnsbl.njabl.org for spam
67.202.57.15 is BLACKLISTED by dnsbl.njabl.org for spam
The following comment is associated with this record: This network is a member of a dynamic hosting environment. See http://ec2.amazonaws.com/
It was added to the list: Tue Apr 1 12:41:39 2008 EST
spam source means the system was found via manual spam header parsing to be the origin of spam.
update july 15th
Agent: firefox/2.0.0.6 (ubuntu-feisty)
72.44.48.95 ec2-72-44-48-95.compute-1.amazonaws.com
Jun 30, 2008
Subscribe to:
Post Comments (Atom)
9 comments:
so how do you ban an amazon bot from accessing your site? What is the coding for it?
I added the domain name to my scripts. Or you could add all of the ips to your IP server block if you can find them all.
tmaster, we'll remove your blog from our index and the visits from our spiders will stop - that's not a problem at all.
Having said that, the reason they are coming in the first place is because someone (perhaps yourself) has subscribed to your RSS feed via our service. Hence, as long as that is ok, we can certainly remove your blog from our system.
Feel free to ping me at ilya _at_ aiderss _dot_ com.
Best,
Ilya
since we use feedburner your bot has no business accessing the blog direct. Also your bot is broken since it can not understand error codes.
BTW its not this blog I was talking about it is the other 100 or so blogs and websites I run.
Tmaster, we also collect additional data like comments - for our PostRank ranking. As I've mentioned before, if you want us to remove your site from our index, please ping me with the list - that is no problem.
Best,
Ilya
Ilya, you really need to honor robots.txt instead of expecting every webmaster to choose to opt out. Robots.txt *is* how we opt out.
And if you don't respect that, you get blocked - which, on sites I host, you have been.
How to ban AmazonAWS.com bots from attacking your web server: http://tonyminhduy.wordpress.com/2009/03/13/how-to-ban-amazonawscom-bots-from-attacking-your-web-server/
How to ban AmazonAWS.com bots from attacking your web server: http://tonyminhduy.wordpress.com/2009/03/13/how-to-ban-amazonawscom-bots-from-attacking-your-web-server/
Post a Comment