A problem appeared on CMAQ where the simplenews subscription module would get used to spam people. In fact, it is probably an old vulnerability that was patched in our instance of it, but bots are still trying to use it, en masse. What this does is that it generates bunch of bounces to the webmaster address. When I got tired of it, i had received more than 250 bounces in a single day.

A solution that works

The current script fixes the problems with the sample code above and generates the right database. It does so by using the BerkeleyDB module.

But first, let's look at how all this ties together.

Apache configuration

homere:/etc/apache# cat conf.d/blacklist.conf
# define a system-wide blacklist database, currently used only on
RewriteMap      blacklist       dbm:/etc/apache/blacklist
# this is the text version, which also works, in case the above fails, which is the case when the database is not in the right format
#RewriteMap     blacklist       txt:/etc/apache/blacklist.txt

Each domain or directory that needs to be protected needs to add a directive like this:

  RewriteCond     ${blacklist:%{REMOTE_ADDR}|OK}  !^OK$
  RewriteRule .* - [F]

/!\ Note that this doesn't work if added straight in a <Directory> section in httpd.conf or included files, I don't know why.

The rewrite rule generates a 403 Forbidden response to the spammers, but it can be changed to redirect to another address or anything rewrite rules can do, basically.

Under the wikifarm

The configuration must sit before any redirection, straight in apache-common.conf:

RewriteEngine   on
RewriteMap      blacklist       dbm:/usr/local/etc/apache2/blacklist.db
RewriteCond     ${blacklist:%{REMOTE_ADDR}|OK}  !^OK$
RewriteRule .* - [F]

## map everything else to moin.cgi/moin.fcg
ScriptAlias / /usr/local/www/wikifarm/server/moin.fcg/

Database maintenance / use cases

The database (/etc/apache/blacklist.db) is generated from it's text version (/etc/apache/blacklist.txt). Both can be used as a Map, but the database version is faster and is currently in use.

Text file format

The text file is a tab-seperated "database". Only the two first fields are used by apache and to generate the .db file. We generally use 5 fields:

  1. an ip address quad
  2. one of "spam", "dos" or "OK", without quotes. "OK" does nothing and the IP is not blocked. Any other value for this field will make the IP be considered as "blocked". We use "spam" and "dos" merely as a quick way to remind ourselves why the IP is blocked, but as long as it's not "OK", the IP is blocked.

  3. a date
  4. notes. we can put here the reason the IP was blocked, in detail: which host, how, the originating country, etc

Adding a new IP to the blacklist

Preferably, the text database is edited, and then the real database is regenerated from that:

homere:/etc/apache# printf "\tspam\t`date`\tcmaq simplenews spam\n" >> blacklist.txt # any editor can be used here (vim, pico, etc)
homere:/etc/apache# /root/bin/txt2dbm blacklist.txt blacklist.db
added key with value spam

The script only mentions added values. Changed values are silently modified in the database. Removed values are not taken into account. The following Makefile eases the burden of having to remember the commandline:

blacklist.db: blacklist.txt
        /root/bin/txt2dbm blacklist.txt blacklist.db

A value can be directly added to the databse using the script, but that is not recommended, since it gets the database out of sync with the flatfile:

homere:/etc/apache# printf "\tspam\t`date`\tcmaq simplenews spam\n" | /root/bin/txt2dbm - blacklist.db
added key with value spam

Removing an IP from the blacklist

Right now, removing an IP is a backward process that requires deleting the database and regenerating it. First, remove the entry in the .txt file, and then:

rm blacklist.db ; touch blacklist.db ; /root/bin/txt2dbm blacklist.txt blacklist.db

That's it.

A solution that doesn't work

One solution was showed to me after I tried a dirty hack to generate a Deny list for the particular domain. My hack was evil and ugly, but would have been quite simple, would Include work in a <Directory> context.

That said, the solution described above is quite elegant, but doesn't work in Apache 1.3 on Sarge. Why? Because the Perl NDBM_File module generates a database that Apache apparently can't read.

CategoryApache CategoryOptimisation

ApacheBlacklisting (last edited 2010-05-06 08:30:31 by MathieuLutfy)