Ban User Agents, referrers, script-kiddies using htaccess
There are many valid reasons to ban a particular request from sucking up your site’s resources; resources that could be better served to valid, interested users. It might be some cross-site attack script, or inward link from a place you don’t want to be associated with, or perhaps a web sucker or download manager, whatever; .htaccess + mod_rewrite provides ways to protect your content from unwanted “guests”.
The basic formula is standard if-then logic: if the request meets a particular CONDITION, then REWRITE the request. The “conditions” can be many things; perhaps the referrer header sent by their browser (the site they came from), or the page they asked for, or a particular query parameter, or the type of client (browser, etc.) they are using, or any other piece of information Apache has attached to the request. Here’s an example which will deny access to “Teleport Pro”, a download manager which is known to suck, hard..
Who need’s a local copy, when I’m right here?..RewriteCond %{HTTP_USER_AGENT} ^Teleport\ Pro [NC]
RewriteRule . abuse.txt [L]
It’s your site, and just like your home, you have every right to exert some control over who gets in. You may have a huge list of user agents you’d rather not have eating your bandwidth; so use the [OR] flag, and line ’em up..
A little garlic for the net vampires..RewriteCond %{HTTP_USER_AGENT} ^BackWeb [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Bandit [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^BatchFTP [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^BecomeBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^BlackWidow [NC,OR]
# etc..
RewriteCond %{HTTP_USER_AGENT} ^Net\ Vampire [NC]
RewriteRule . abuse.txt [L]
This forms the basis of what often becomes a HUGE list of ban-lines. Remember, we aren’t limited to user agent strings..
Suckers, h4x0rz, kiddies, cross-site scripters and more.. Bye now!# why not come visit me directly?
RewriteCond %{HTTP_REFERER} \.opendirviewer\. [NC,OR]
# this prevents stoopid cross-site discovery attacks..
RewriteCond %{THE_REQUEST} \?\ HTTP/ [NC,OR]
# please stop pretending to be the Googlebot..
RewriteCond %{HTTP_REFERER} users\.skynet\.be.* [NC,OR]
# really, we need a special page for these twats..
RewriteCond %{QUERY_STRING} \=\|w\| [NC,OR]
RewriteCond %{THE_REQUEST} etc/passwd [NC,OR]
RewriteCond %{REQUEST_URI} owssvr\.dll [NC,OR]
# you can probably work these out..
RewriteCond %{QUERY_STRING} \=\|w\| [NC,OR]
RewriteCond %{THE_REQUEST} \/\*\ HTTP/ [NC,OR]
# etc..
RewriteCond %{HTTP_USER_AGENT} Sucker [NC]
RewriteRule . abuse.txt [L]
Fortunately, mod_rewite can parse enormous lists of ban-lines in milliseconds, so feel free to be as specific and comprehensive as required.
As ever, thorough testing is strongly recommended. Simply send requests matching your conditions and see what happens. And importantly; normal requests, too. Firefox, Opera, Konqueror, and most other decent browsers, allow you to alter the user agent string; though you would quickly find the process tedious in a testing situation. Far better to use some tool better designed to send fake HTTP requests..
It’s not too difficult to mock up a web request on the command-line with any-old user agent using a scripting language like php or Perl, if you have these things available (read: most Linux/UNIX/BSD/etc. as well as many other OS). Many examples exist online. In fact, you could quickly create a suite of tests, designed to interrogate all your rewrite rules, with results logging and much more, if required. cURL is always useful for jobs like this, so long as you don’t add a cURL ban-line!
On a Windows desktop, Sam Spade can send a single spoofed request with a couple of clicks, along with a stack of similarly handy tricks, and regularly proves itself invaluable.
Parmendra's PHP Blog 