Ban User Agents and referrers and script-kiddies through htaccess

Ban User Agents, referrers, script-kiddies using htaccess

There are many valid reasons to ban a particular request from sucking up your site’s resources; resources that could be better served to valid, interested users. It might be some cross-site attack script, or inward link from a place you don’t want to be associated with, or perhaps a web sucker or download manager, whatever; .htaccess + mod_rewrite provides ways to protect your content from unwanted “guests”.

The basic formula is standard if-then logic: if the request meets a particular CONDITION, then REWRITE the request. The “conditions” can be many things; perhaps the referrer header sent by their browser (the site they came from), or the page they asked for, or a particular query parameter, or the type of client (browser, etc.) they are using, or any other piece of information Apache has attached to the request. Here’s an example which will deny access to “Teleport Pro”, a download manager which is known to suck, hard..

Who need’s a local copy, when I’m right here?..

RewriteCond %{HTTP_USER_AGENT} ^Teleport\ Pro [NC]
RewriteRule . abuse.txt [L]

It’s your site, and just like your home, you have every right to exert some control over who gets in. You may have a huge list of user agents you’d rather not have eating your bandwidth; so use the [OR] flag, and line ’em up..

A little garlic for the net vampires..

RewriteCond %{HTTP_USER_AGENT} ^BackWeb [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Bandit [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^BatchFTP [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^BecomeBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^BlackWidow [NC,OR]
# etc..
RewriteCond %{HTTP_USER_AGENT} ^Net\ Vampire [NC]
RewriteRule . abuse.txt [L]

This forms the basis of what often becomes a HUGE list of ban-lines. Remember, we aren’t limited to user agent strings..

Suckers, h4x0rz, kiddies, cross-site scripters and more.. Bye now!

# why not come visit me directly?
RewriteCond %{HTTP_REFERER} \.opendirviewer\. [NC,OR]
# this prevents stoopid cross-site discovery attacks..
RewriteCond %{THE_REQUEST} \?\ HTTP/ [NC,OR]
# please stop pretending to be the Googlebot..
RewriteCond %{HTTP_REFERER} users\.skynet\.be.* [NC,OR]
# really, we need a special page for these twats..
RewriteCond %{QUERY_STRING} \=\|w\| [NC,OR]
RewriteCond %{THE_REQUEST} etc/passwd [NC,OR]
RewriteCond %{REQUEST_URI} owssvr\.dll [NC,OR]
# you can probably work these out..
RewriteCond %{QUERY_STRING} \=\|w\| [NC,OR]
RewriteCond %{THE_REQUEST} \/\*\ HTTP/ [NC,OR]
# etc..
RewriteCond %{HTTP_USER_AGENT} Sucker [NC]
RewriteRule . abuse.txt [L]

Fortunately, mod_rewite can parse enormous lists of ban-lines in milliseconds, so feel free to be as specific and comprehensive as required.

As ever, thorough testing is strongly recommended. Simply send requests matching your conditions and see what happens. And importantly; normal requests, too. Firefox, Opera, Konqueror, and most other decent browsers, allow you to alter the user agent string; though you would quickly find the process tedious in a testing situation. Far better to use some tool better designed to send fake HTTP requests..

It’s not too difficult to mock up a web request on the command-line with any-old user agent using a scripting language like php or Perl, if you have these things available (read: most Linux/UNIX/BSD/etc. as well as many other OS). Many examples exist online. In fact, you could quickly create a suite of tests, designed to interrogate all your rewrite rules, with results logging and much more, if required. cURL is always useful for jobs like this, so long as you don’t add a cURL ban-line!

On a Windows desktop, Sam Spade can send a single spoofed request with a couple of clicks, along with a stack of similarly handy tricks, and regularly proves itself invaluable.

cooler access denied through htaccess

cooler access denied using htaccess

The trouble with this is the way our user gets a 403 “Access Denied” error, which is a bit like having a door slammed in your face. Fortunately, mod_rewrite comes to the rescue again and enables us to do less painful things. One method I often employ is to redirect the user to the parent folder..

they go “huh?.. ahhh!”

# send them up!
Options +FollowSymlinks
RewriteEngine on
RewriteRule ^(.*)$ ../ [NC]

It works great, though it can be a wee bit tricky with the URLs, and you may prefer to use a harder location, which avoids potential issues in indexed directories, where folks can get in a loop..

they go damn! Oh!

# send them exactly there!
Options +FollowSymlinks
RewriteEngine on
RewriteRule ^(.*)$ /comms/hardware/router/ [NC]

Sometimes you’ll only want to deny access to most of the files in the directory, but allow access to maybe one or two files, or file types, easy..
deny with style!

# users can load only “special.zip”, and the css and js files.
Options +FollowSymlinks
RewriteEngine On
RewriteCond %{REQUEST_FILENAME} !^(.+)\.css$
RewriteCond %{REQUEST_FILENAME} !^(.+)\.js$
RewriteCond %{REQUEST_FILENAME} !special.zip$
RewriteRule ^(.+)$ /chat/ [NC]

Here we take the whole thing a stage further. Users can access .css (stylesheet) and javascript files without problem, and also the file called “special.zip”, but requests for any other filetypes are immediately redirected back up to the main “/chat/” directory. You can add as many types as you need. You could also bundle the filetypes into one line using | (or) syntax, though individual lines are perhaps clearer.

Here’s what’s currently cooking inside my /inc/ directory..

all-in-one control..

RewriteEngine on
Options +FollowSymlinks
# allow access with no restrictions to local machine at 192.168.1.3
RewriteCond %{REMOTE_ADDR} !192.168.1.3
# allow access to all .css and .js in sub-directories..
RewriteCond %{REQUEST_URI} !\.css$
RewriteCond %{REQUEST_URI} !\.js$
# allow access to the files inside img/, but not a directory listing..
RewriteCond %{REQUEST_URI} !img/(.*)\.
# allow access to these particular files…
RewriteCond %{REQUEST_URI} !comments.php$
RewriteCond %{REQUEST_URI} !parammail.php$
RewriteCond %{REQUEST_URI} !digitrack.php$
RewriteCond %{REQUEST_URI} !gd-verify.php$
RewriteCond %{REQUEST_URI} !post-dumper.php$
RewriteCond %{REQUEST_URI} !print.php$
RewriteCond %{REQUEST_URI} !source-dump.php$
RewriteCond %{REQUEST_URI} !textview.php$
RewriteRule ^(.*)$ / [R,NC,L]

capturing variables through htaccess

Slapping (.*) onto the end of the request part of a ReWriteRule is just fine when using a simple $_GET variable, but sometimes you want to do trickier things, like capturing particular variables and converting them into other variables in the target URL. Or something else..

When capturing variables, the first thing you need to know about, is the [QSA] flag, which simply tags all the original variables back onto the end of the target url. This may be all you need, and will happen automatically for simple rewites. The second thing, is %{QUERY_STRING}, an Apache server string we can capture variables from, using simple RewriteCond (aka. conditional ) statements.

RewriteCond is very like doing if...then...do in many programming languages. If a certain condition is true, then do the rewrite that follows..

In the following example, the RewriteCond statement checks that the query string has the foo variable set, and captures its value while it’s there. In other words, only requests for /grab that have the variable foo set, will be rewritten, and while we’re at it, we’ll also switch foo, for bar, just because we can..

capturing a $_GET variable:

Options +FollowSymlinks
RewriteEngine On
RewriteCond %{QUERY_STRING} foo=(.*)
RewriteRule ^grab(.*) /page.php?bar=%1

would translate a link/user’s request for..

http://domain.com/grab?foo=bar

server-side, into..

http://domain.com/page.php?bar=bar

Which is to say, the user’s browser would be fed page.php (without an [R] flag in the RewriteRule, their address bar would still read /grab?foo=bar). The variable bar would be available to your script, with its value set to bar. This variable has been magically created, by simply using a regular ? in the target of the RewriteRule, and tagging on the first captured backreference, %1.. ?bar=%1

Note how we use the % character, to specify variables captured in RewriteCond statements, aka “Backreferences”. This is exactly like using $1 to specify numbered backreferences captured in RewriteRule patterns, except for strings captured inside a RewriteCond statement, we use % instead of $. Simple.

You can use the [QSA] flag in addition to these query string manipulations, merge them. In the next example, the value of foo becomes the directory in the target URL, and the variable file is magically created. The original query string is then tagged back onto the end of the whole thing..

QSA Overkill!

Options +FollowSymlinks
RewriteEngine On
RewriteCond %{QUERY_STRING} foo=(.+)
RewriteRule ^grab/(.*) /%1/index.php?file=$1 [QSA]

So a request for..

http://domain.com/grab/foobar.zip?level=5&foo=bar

is translated, server-side, into..

http://domain.com/bar/index.php?file=foobar.zip&level=5&foo=bar

Depending on your needs, you could even use flat links and dynamic variables together, something like this could be useful..

mixing flat and dynamic links in a single ruleset..

Options +FollowSymlinks
RewriteEngine On
RewriteCond %{QUERY_STRING} version=(.+)
RewriteRule ^grab/([^/]+)/(.*) /%1/index.php?section=$1&file=$2 [QSA]

By the way, you can easily do the opposite, strip a query string from a URL, by simply putting a ? right at the end of the taget part. This example does exactly that, whilst leaving the actual URI intact..

just a demo!

Options +FollowSymlinks
RewriteEngine On
RewriteCond %{QUERY_STRING} .
RewriteRule foo.php(.*) /foo.php? [L]

The RewriteCond statement only allows requests that have something in their query string, to be processed by the RewriteRule, or else we’d end up in that hellish place, dread to all mod_rewriters.. the endless loop. RewriteCond is often used like this; as a safety-net.