Installation simple de spamassassin avec les utilisatteurs dans mySQL sur Debian Jessie (sous AtlernC)

Pour 13258 et scanner.koumbit.net.

Aussi, réalisé ici: 14248 pour un client.

Aussi, je teste la version de debian backport dans jessie 24938

Configuration de base

Installer le package spamassassin:

aptitude install spamassassin libmail-dkim-perl razor pyzor

Ajouter ces options à /etc/default/spamassassin:

CRON=1
OPTIONS="--create-prefs --max-children 5 --helper-home-dir --listen localhost --nouser-config --sql-config --allow-tell --username debian-spamd"

--allowed-ips=127.0.0.1 a été retiré parce que parfois le localhost ne peut pas connecter...

Si vous désirez avoir spamassassin qui écoute sur un interface réseau spécifique, ajouter le paramètre --listen-ip=XXX.XXX.XXX.XXX

Rouler la commande suivante pour activer le service au démarrage.

# systemctl enable spamassassin.service

Ceci créé un serveur spamassassin qui écoute sur l'interface locale (ajouter d'autres --allowed-ips pour permettre à d'autres hosts de se connecter). Le client spamc peut alors s'y connecter pour filtrer des courriels. --allow-tell est pour permettre l'apprentissage. Les autres options sont documentées dans la manpage de spamd. L'option --username debian-spamd est nécessaire sinon, ça donne l'erreur suivante: spamd: cannot use --sql-config without -u

Éditer le fichier, /etc/spamassassin/local.cf pour permettre ajouter la mention ***SPAM*** des courriels qui sont détecté comme pourriel. report_safe 0 permet d'éviter d'avoir un long rapport de spamassassin dans le corps du courriel, il ajoutera quelques lines à l'enveloppe de courriel.

rewrite_header Subject ***SPAM***
report_safe 0

Configuration MySQL

Créer un user et une database MySQL pour le daemon:

CREATE DATABASE spamassassin;
GRANT INSERT,UPDATE,DELETE,SELECT ON spamassassin.* TO 'spamassassin'@'%' IDENTIFIED BY 'XXXXXXXX';

Configurer /etc/spamassassin/sql.cf avec les users mysql:

user_scores_dsn               DBI:mysql:spamassassin:localhost
user_scores_sql_username      spamassassin
user_scores_sql_password      XXXXXXXXXXXX

auto_whitelist_factory Mail::SpamAssassin::SQLBasedAddrList

user_awl_dsn               DBI:mysql:spamassassin:localhost
user_awl_sql_username      spamassassin
user_awl_sql_password      XXXXXXXXXXXX
user_awl_sql_table         awl

bayes_store_module                 Mail::SpamAssassin::BayesStore::SQL

bayes_sql_dsn                      DBI:mysql:spamassassin:localhost
bayes_sql_username                 spamassassin
bayes_sql_password                 XXXXXXXXXXXX

Protéger le fichier de config:

chmod 640 sql.cf
chgrp debian-spamd sql.cf

Ensuite, charger les schémas de base de données:

sed 's/ TYPE=MyISAM//' < /usr/share/doc/spamassassin/sql/*mysql.sql | mysql spamassassin

Le sed est nécessaire sinon on a l'erreur suivante sur un install par défaut de MySQL dans Debian Wheezy:

ERROR 1064 (42000) at line 1: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'TYPE=MyISAM' at line 8

application des nouvelles configurations

redémarrage du service spamassassin

service spamassassin restart

Différents tests

Tester si le daemon fonctionne bien:

spamc -u test@localhost --dest=localhost < /usr/share/doc/spamassassin/examples/sample-spam.txt
gunzip -c /usr/share/doc/spamassassin/examples/sample-nonspam.txt.gz | spamc -u test@locahost --dest=localhost

--dest=localhost est nécessaire, sinon on obtient

 spamd: unauthorized connection from ::1 [::1]:48290 to port 783, fd 6 at /usr/sbin/spamd line 1580.

Essayer d'envoyer un courriel pour confirmer qu'il est bien identifié comme "spam"

On veut le sujet suivant:

XJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-UBE-TEST-EMAIL*C.34X

spamassassin devrait l'identifier comme spam.

2 autres méthodes pour faire des tests ici: https://easyengine.io/tutorials/mail/server/testing/spam/

Déployer avec postfix milter

  1. Installer spamass-milter:

    apt install spamass-milter
  2. Ajouter le path de connection à smtpd_milters dans main.cf:

    non_smtpd_milters = $non_smtpd_milters
    smtpd_milters = unix:/var/spool/postfix/spamass/spamass-milter.sock # inet:127.0.0.1:8891 # (dkim)

Pourquoi? Le spamassassin-milter est en pont entre postfix et spamd qui nous permet d'utiliser des checks milter (milter_header_checks) pour traiter les messages d'un manière plus costaud, eg. rejeter tout ce qui à un score >= XXX etc.

N.B Si tout l'entrainement sur le serveur est sur un utilisateur, eg debian-spamd, ajouter

OPTIONS="${OPTIONS} -- -u debian-spamd"

à la fin de /etc/default/spamass-milter

N.B. C'est pas tant important d'avoir spamassassin dans le non_smtpd_milters. Voir les docs de postfix sure où les milters sont appliqués

Déployer avec postfix "content filter"

On ajout les lignes suivantes au fichier /etc/postfix/master.cf

--- a/master.cf
+++ b/master.cf
@@ -14,6 +14,10 @@
 #               (yes)   (yes)   (yes)   (never) (100)
 # ==========================================================================
 smtp      inet  n       -       -       -       -       smtpd
+  -o content_filter=spamassassin
+spamassassin    unix    -       n       n       -       -       pipe
+  user=debian-spamd argv=/usr/bin/spamc --dest=localhost -f -e
+  /usr/sbin/sendmail -oi -f ${sender} ${recipient}
 submission inet n       -       -       -       -       smtpd

redémarrage du service postfix

service postfix restart

Vérfier les enveloppes des courriels, vous avez aller maintenant avoir des nouvelles lignes

X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on hostname
X-Spam-Level: **
X-Spam-Status:

voir les logs...

# tail -f /var/log/syslog | grep spamd

Si postfix utilise DNS(DNSBL) de spamhaus ...

Spamassassin par défaut fait aussi une vérification sur le service de spamhaus, on ne veut pas dupliquer la tâche, on va le désactiver dans spamassassin et laisser postfix faire le boulot. Ça semble relatif au désir des clients. On note que spamhaus fait un premier ménage pour pas qu'on traite trop de spam par spamassassin, ensuite spamassassin fait des vérifications avec d'autres services de DNSBL donc, on veut garder la vérification active dans spamassassin.

Déployer avec .forward/procmail

spamc peut être intégré à un fichier .forward, à procmail

Exemple pour RT:

# file managed through puppet

:0fw: spamassassin.lock
* < 300000
| /usr/bin/spamc -u rt@rt.example.net

#if the spam trigger is fired send to spam queue
:0w
* ^(X-Spam-Status: Yes|X-DSPAM-Result: Spam)
{
        QUEUE=Spam
}

Exemple procmail:

:0
* ^X-Spam-Status: Yes
.junk/

À considérer

# loadplugin Mail::SpamAssassin::Plugin::Rule2XSBody

... dans /etc/spamassassin/v320.pre

pour les pourriels en d'autres langues...

https://spamassassin.apache.org/full/3.2.x/doc/Mail_SpamAssassin_Conf.html

diff --git a/spamassassin/v310.pre b/spamassassin/v310.pre
index 56e5b2f..55f3e83 100644
--- a/spamassassin/v310.pre
+++ b/spamassassin/v310.pre
@@ -50,7 +50,7 @@ loadplugin Mail::SpamAssassin::Plugin::AutoLearnThreshold
 
 # TextCat - language guesser
 #
-#loadplugin Mail::SpamAssassin::Plugin::TextCat
+loadplugin Mail::SpamAssassin::Plugin::TextCat

diff --git a/spamassassin/local.cf b/spamassassin/local.cf
index 7c3407a..9ecd86c 100644
--- a/spamassassin/local.cf
+++ b/spamassassin/local.cf
@@ -7,6 +7,15 @@
 #
 ###########################################################################
 
+# conf comme demandé ici : https://redmine.koumbit.net/issues/12087
+# language config, requires Mail::SpamAssassin::Plugin::TextCat
+ok_languages de en es fr pt
+
+# demandé https://redmine.koumbit.net/issues/12087
+# comme documenté
+# https://spamassassin.apache.org/full/3.2.x/doc/Mail_SpamAssassin_Conf.html
+ok_locales en
+

activer le suivi de spamassassin par munin

On veut pouvoir la quantité de détection de spam par spamassassin. Rien de mieux qu'un plugin munin, le plugin est disponible dans le package munin-plugins-extra

Si on veut l'ajouter en utilisant puppet,

    munin::plugin { 'spamstats':
      ensure => present,
      config => "group adm\n";
    }

Déplacer les spams dans le répertoire Junk (avec Dovecot et sieve)

Dans la version AlternC 3.2.10, on a Dovecot et le filtrage des courriels avec Sieve. Nous allons utiliser une règle global dans sieve pour déplacer un courriel détecté dans le répertoire Junk de l'utilisateur. Vérifier si le répertoire Junk ou INBOX.Junk existe.

mkdir /etc/dovecot/sieve/
cat >> /etc/dovecot/sieve/default.sieve << EOF
require "fileinto";
if header :contains "X-Spam-Flag" "YES" {
  fileinto "Junk";
}
EOF

On compile le filtre

sievec /etc/dovecot/sieve/default.sieve

On ajuste les permissions:

chown -R vmail:vmail /etc/dovecot/sieve/

Ajouter la configuration à dovecot pour activer les filtres globaux et qui doivent être utilisé avant les filtres personalisés.

Au fichier suivant: /etc/dovecot/conf.d/95_alternc.conf

plugin {
....

  sieve_global_path = /etc/dovecot/sieve/default.sieve
  sieve_global_dir = /etc/dovecot/sieve/
  sieve_before = /etc/dovecot/sieve/
}

débug

Si vous activé le débug dans dovecot, vous allez voir une ligne comme ça dans mail.log, si ça fonctionne bien.

Debug: sieve: script binary /etc/dovecot/sieve/default.svbin successfully loaded

Ensuite, sans le débug en mode normal, vous allez retrouver une ligne comme ça quand le filtre s'applique.

stored mail into mailbox 'INBOX.Junk'

reférence: http://wiki1.dovecot.org/LDA/Sieve/Dovecot

Activer les plugins dans roundcube

Activer le plugin "markasjunk" dans la conf de roundcube, dans le fichier /etc/roundcube/config.inc.php

:$rcmail_config['plugins'] = array("managesieve","password","markasjunk");

markasjunk2 semble vraiment intéressant mais il n'est pas distribué dans jessie, on va attendre strech.

Postfix simple content filter configuration

Following the postfix integration guide, I have performed the following surgery:

--- master.cf.orig      Tue Apr 18 16:27:30 2006
+++ master.cf   Tue Apr 18 16:27:31 2006
@@ -6,7 +6,7 @@
 # service type  private unpriv  chroot  wakeup  maxproc command + args
 #               (yes)   (yes)   (yes)   (never) (100)
 # ==========================================================================
-smtp      inet  n       -       n       -       -       smtpd
+smtp      inet  n       -       n       -       -       smtpd -o content_filter=spamassassin
 #submission inet n      -       n       -       -       smtpd
 #      -o smtpd_etrn_restrictions=reject
 #      -o smtpd_client_restrictions=permit_sasl_authenticated,reject
@@ -77,3 +77,5 @@
   flags=F user=ftn argv=/usr/lib/ifmail/ifmail -r $nexthop ($recipient)
 bsmtp     unix  -       n       n       -       -       pipe
   flags=Fq. user=foo argv=/usr/local/sbin/bsmtp -f $sender $nexthop $recipient
+spamassassin
+          unix  -       n       n       -       -       pipe user=postfix argv=/usr/local/bin/spamc -u ${recipient} -e /usr/sbin/sendmail -oi -f ${sender} ${recipient}

This way postfix pipes all incoming mails through spamc.

alternc.fqccl.org specifics

/etc/procmailrc

UMASK=007
RECIPIENT=$1
USR=`echo ${RECIPIENT} | grep -o '^[^@]*'`
LETTER=`echo ${USR} | head -c 1`
MAILDIR=/var/alternc/mail/$LETTER/$USR/Maildir
DEFAULT=${MAILDIR}/

:0
* ^X-Spam-Status: Yes
${MAILDIR}/.spam/

master.cf

'procmail' and 'spamassassin' transports added in /etc/postfix/master.cf, filter added as per above (user=spamassassin) :

procmail  unix  -       n       n       -       -       pipe
  flags=Rhu user=www-data argv=/usr/bin/procmail -a ${recipient} -p /etc/procmailrc
spamassassin
  unix  -       n       n       -       -       pipe user=spamassassin argv=/usr/bin/spamc -e /usr/sbin/sendmail -oi -f ${sender} ${recipient}

main.cf

default transport changed from 'local' to 'procmail' in /etc/postfix/main.cf:

local_transport = procmail

We also need to tell postfix not to deliver too many emails to procmail at the same time, since it only knows how to deal with one at a time:

procmail_destination_recipient_limit = 1

/usr/local/sbin/expire_spam

find /var/alternc/mail -type f -mtime +7 -iregex '^.*Maildir/\.spam/.*' -exec rm -v {} \;

/etc/crontab

11 3    * * *   www-data /usr/local/sbin/expire_spam

Advanced content filter with Amavis

The above works well for small sites, but doesn't degenerate gracefully, as it will bounce mail when out of memory or resources, for example. The solution is to setup a content filter using Amavis, as described in the FILTER_README.

Quick recall, the general principle is to have a parallel SMTP server (Amavis) that is designed to accept mail to be scanned, and send it back (or not) into the mailserver. Here is a general picture to help us remember what port numbers we're talking about here:

+----------+    +-------+    +--------------+    +-------------+
| smtpd:25 | -> | queue | -> | amavis:10024 | -> | smtpd:10025 |
+----------+    +-------+    +--------------+    +-------------+
                   ^                                    |
                   |                                    |
                   +------------------------------------+

smtpd:25 is the regular SMTP server that takes incoming mail. queue is the regular postfix queue. amavis:10024 is the amavis mailserver, running on port 10024, and smtpd:10025 is a special postfix SMTP server that takes filtered mail and reinjects it in the queue properly (so that it doesn't get filtered again, for example).

Amavis setup

(aptitude install amavisd-new...)

We setup amavis to listen on eth1 for 2 reasons: (1) it doesn't seem to want to do it on 127.0.0.1, probably because of the vserver, and (2) because it can be used by other servers on the internal network later on.

The code below is all in /etc/amavis/amavisd.conf.

$inet_socket_bind = '192.168.0.104';
@inet_acl = qw( 127.0.0.1  192.168.0.104 );

$forward_method = 'smtp:192.168.0.104:10025';  # where to forward checked mail
$notify_method = $forward_method;          # where to submit notifications

Don't modify subject if not scanned:

$undecipherable_subject_tag = undef;

Spamassassin setup in Amavis

Because some settings are overridden by amavis (??), they need to be done in /etc/amavis/amavisd.conf:

$sa_tag_level_deflt  = undef; # always tag messages
$sa_tag2_level_deflt = 5; # add 'spam detected' headers at that level
$sa_kill_level_deflt = $sa_tag2_level_deflt; # triggers spam evasive actions
                           # at or above that level: bounce/reject/drop,
                           # quarantine, and adding mail address extension

$sa_dsn_cutoff_level = 10;  # spam level beyond which a DSN is not sent,
                            # effectively turning D_BOUNCE into D_DISCARD;
                            # undef disables this feature and is a default

# a quick reference:
#   tag_level  controls adding the X-Spam-Status and X-Spam-Level headers,
#   tag2_level controls adding 'X-Spam-Flag: YES', and editing Subject,
#   kill_level controls 'evasive actions' (reject, quarantine, extensions);
# it only makes sense to maintain the relationship:
# tag_level <= tag2_level <= kill_level < $sa_dsn_cutoff_level

Since we treat evil spam elsewhere (in procmail), we let it pass through amavis:

$final_virus_destiny      = D_DISCARD; # (defaults to D_BOUNCE)
$final_banned_destiny     = D_BOUNCE;  # (defaults to D_BOUNCE)
$final_spam_destiny       = D_PASS;  # (defaults to D_REJECT)
$final_bad_header_destiny = D_PASS;  # (defaults to D_PASS), D_BOUNCE suggested

... and don't quarantine it:

$spam_quarantine_to = undef;

In /etc/spamassassin/local.cf, we configure more stuff, in particular:

#   Save spam messages as a message/rfc822 MIME attachment instead of
#   modifying the original message (0: off, 2: use text/plain instead)
report_safe 1

chroot

Voir README.chroot.txt pour l'implantation, pas l'air si simple...

Accepting mail from the content filter

In /etc/postfix/master.cf:

192.168.0.104:10025 inet  n       -       n       -       10      smtpd
        -o content_filter= 
        -o receive_override_options=no_unknown_recipient_checks,no_header_body_checks
        -o smtpd_helo_restrictions=
        -o smtpd_client_restrictions=
        -o smtpd_sender_restrictions=
        -o smtpd_recipient_restrictions=permit_mynetworks,reject
        -o mynetworks=192.168.0.0/24
        -o smtpd_authorized_xforward_hosts=192.168.0.104

don't forget a postfix reload.

First tests

You can right now test if amavis can receive mail and send it back into postfix:

swaks -t anarcat@koumbit.org -s 192.168.0.104 -p 10024

Basically, using this, we're able to test the filtering capacities without hooking them into the main queue and SMTP server, by talking directly to Amavis.

Antivirus (ClamAv) setup

We start by leaving Clamav disabled:

@av_scanners = ();
@av_scanners_backup =();

This should put after the default definitions so that no AV is defined and therefore tried.

Spamassassin setup

aptitude install spamassassin. well, not exactly, we install the version from backports.org using the GUI. we use release 3.1 because of the following features:

We remove spamc because we do not use it.

To activate spamassassin in amavis, we comment out the line:

#@bypass_spam_checks_acl  = qw( . );    # No default dependency on spamassassin

in amavisd.conf and of course restart amavis.

Final deployment

check if it's not better to use amavis' quarantine instead of procmail for isolating spam...

These are the steps currently planned in the deployment:

  1. Notify users (day 0) (./)

  2. Activate filtering in postfix (day 0) (./)

  3. Hook in monitors (day 0) (./)

  4. Test performance (day 0)
  5. Activate procmail filters & renotify users (day 7)

  6. Consider Clamav
  7. Consider learning capabilities

Notify users

Users will have to be notified of the new service and its impacts. Important facts:

  1. mail will initially not be filtered, only tagged, for 1 week
  2. when filtered away, it will be stored in a junk subfolder, available through IMAP or webmail
  3. one month (28 days) old email in the junk folder will be deleted each night
  4. spam-learning and virus scanning might be deployed in the near future

/FirstNotification was sent today. -- TheAnarcat 2006-08-14 10:51:57

Activate filtering in postfix

In master.cf:

scan      unix  -       -       n       -       2      smtp
        -o smtp_send_xforward_command=yes
        -o disable_mime_output_conversion=yes

The "2" there is the process limit, in sync with amavis.

In main.cf:

content_filter = scan:192.168.0.104:10024
receive_override_options = no_address_mappings

Hook in monitors

See MuninService/SpamassassinPlugin.

Test performance

Look for differences in deliveries delays. Compare those two graphs:

http://stats.koumbit.net/koumbit.net/homere.koumbit.net-amavis.html http://stats.koumbit.net/koumbit.net/romulus.koumbit.net-load.html

Activate procmail filters & renotify users

Consider the alternc-procmail package to allow people to create their own rules, while we're at it.

This consists mostly in duplicating the procmail setup for alternc.fqccl.org described above. This includes setting up the cronjob, but for 28-days checks.

Also consider the capabilities of postfix and amavis regarding address extensions: amavis could just forward spam email to foo+junk@example.com instead of letting procmail refilter the mail.

Consider Clamav

Test and deploy an antivirus.

Consider learning capabilities

See if it's possible to have amavis take care of storing preferences per-recipient instead of per-UNIX user, as is usually the case with spamc/spamd.

performance analysis

  1. amavis is said to be at least as fast as spamc/spamd because it loads spamassassin internally
  2. the "layered" approach allows for easily offloading the service to another host
  3. <!> check if spam checks are done for non-existent emails

  4. check of compliance with requesites above

With spampd

Seems like a simple enough configuration:

apt-get install spampd
vi /etc/postfix/master.cf
postfix reload

Modifications required to master.cf:

smtp      inet  n       -       -       -       -       smtpd
        -o content_filter=smtp:127.0.0.1:10025
        -o myhostname=mx.anarcat.ath.cx
localhost:10026        inet    n       -       n       -       10      smtpd
        -o content_filter=
        -o myhostname=delivery.anarcat.ath.cx

... that's all! spampd talks with spamassassin and uses the normal configuration.

Above config out of date, used by anarcat on his home setup, see http://anarc.at/hardware/server/marcos/configuration/?updated#index1h4 for details.

Simple lethe / FreeBSD install notes

That was a pain. After the perl 5.8 upgrade, I had to reinstall it, but then it was dependent upon p5-Net, itself on p5-Authen-SASL and then p5-GSSAPI which didn't compile, probably because I didn't have kerberos compiled on the machine. I commented the dependency in p5-Authen-SASL and all was merry.

Démarrer le filtre Bayesien

Voir: SpamAssassinMaintenance#Apprentissage_Bayesien

Original specifications

(!) required for FQCCL

  1. don't filter for non-existent accounts (!)

  2. junk must go in a junk subfolder (!)

  3. virus detection must be included with Amavisd, which should probably be taking care of talking to spamassassin
  4. junk mail is deleted after 4 weeks (7 days for fqccl) (ie. >= 4 weeks old junk mail is deleted each night) (!)

  5. bayesian filtering and spamassassin preferences either
    1. per email addresses (will require hacking because spamc -u requires a real username)

    2. per alternc account (will also require hacking)
  6. learning can be done by putting spam in a spam subfolder and ham in a ham subfolder
  7. MuninService/SpamassassinPlugin setup to graph spam detection

  8. don't affect too much performance (!)

Voir PropositionSpamAssassinAmavis.

References

SpamAssassinConfiguration (last edited 2019-05-16 14:33:08 by kienan)