robots.txt Tutorial - Block Bad Bots
Some bots will ignore robots.txt files as they don't care if you want
them on your web site or not.
These can be blocked by using a .htaccess file instead.
1. Block robots via .htaccess
We can't block by robot name here, we block them by matching the beginning of their User-Agent string.
SetEnvIfNoCase User-Agent "^EmailSiphon" bad_bot|
SetEnvIfNoCase User-Agent "^EmailWolf" bad_bot
SetEnvIfNoCase User-Agent "^ExtractorPro" bad_bot
SetEnvIfNoCase User-Agent "^CherryPicker" bad_bot
SetEnvIfNoCase User-Agent "^NICErsPRO" bad_bot
SetEnvIfNoCase User-Agent "^Teleport" bad_bot
SetEnvIfNoCase User-Agent "^EmailCollector" bad_bot
<Limit GET POST>
Allow from all
Deny from env=bad_bot
This example bans robots on our list of spambots.
To block another robot, add a line for it near the top.
SetEnvIfNoCase User-Agent "^User-Agent" bad_bot|
Replace User-Agent with the User-Agent string for this robot, as found
in log files. Here's a sample log entry.
xyz.net - - [07/Mar/2003:11:28:35] "GET / HTTP/1.0" 403 - "-" "Teleport 1.28"
Here, the User-Agent is Teleport 1.28. The ^
character in the SetEnvIfNoCase lines tells our .htaccess file to block anything
starting with the string we provide.
Any User-Agent starting directly with Teleport would be blocked, regardless
of version number or added text.
2. Tool to create the .htaccess file
This tool can create .htaccess files for you, blocking some of the robots
discussed in this tutorial.
You can also enter up to six custom User-Agent strings to have blocked from
your site. Enter one per box.