Le Wiki Koumbit
Contents
-
Installation simple de spamassassin avec les utilisatteurs dans mySQL sur Debian Jessie (sous AtlernC)
- Configuration de base
- Configuration MySQL
- application des nouvelles configurations
- Différents tests
- Déployer avec postfix milter
- Déployer avec postfix "content filter"
- Si postfix utilise DNS(DNSBL) de spamhaus ...
- Déployer avec .forward/procmail
- À considérer
- activer le suivi de spamassassin par munin
- Déplacer les spams dans le répertoire Junk (avec Dovecot et sieve)
- Postfix simple content filter configuration
- Advanced content filter with Amavis
- With spampd
- Simple lethe / FreeBSD install notes
- Démarrer le filtre Bayesien
- Original specifications
- References
Installation simple de spamassassin avec les utilisatteurs dans mySQL sur Debian Jessie (sous AtlernC)
Pour 13258 et scanner.koumbit.net.
Aussi, réalisé ici: 14248 pour un client.
Aussi, je teste la version de debian backport dans jessie 24938
Configuration de base
Installer le package spamassassin:
aptitude install spamassassin libmail-dkim-perl razor pyzor
- On veut libmail-dkim-perl, sinon spamassassin arrive pas a vérifier les signatures dkim, ça va toujours donner "T_DKIM_INVALID"
- razor pyzor sont apparament 2 très bons plugins... à voir.
Ajouter ces options à /etc/default/spamassassin:
CRON=1 OPTIONS="--create-prefs --max-children 5 --helper-home-dir --listen localhost --nouser-config --sql-config --allow-tell --username debian-spamd"
--allowed-ips=127.0.0.1 a été retiré parce que parfois le localhost ne peut pas connecter...
Si vous désirez avoir spamassassin qui écoute sur un interface réseau spécifique, ajouter le paramètre --listen-ip=XXX.XXX.XXX.XXX
Rouler la commande suivante pour activer le service au démarrage.
# systemctl enable spamassassin.service
Ceci créé un serveur spamassassin qui écoute sur l'interface locale (ajouter d'autres --allowed-ips pour permettre à d'autres hosts de se connecter). Le client spamc peut alors s'y connecter pour filtrer des courriels. --allow-tell est pour permettre l'apprentissage. Les autres options sont documentées dans la manpage de spamd. L'option --username debian-spamd est nécessaire sinon, ça donne l'erreur suivante: spamd: cannot use --sql-config without -u
Éditer le fichier, /etc/spamassassin/local.cf pour permettre ajouter la mention ***SPAM*** des courriels qui sont détecté comme pourriel. report_safe 0 permet d'éviter d'avoir un long rapport de spamassassin dans le corps du courriel, il ajoutera quelques lines à l'enveloppe de courriel.
rewrite_header Subject ***SPAM*** report_safe 0
Configuration MySQL
Créer un user et une database MySQL pour le daemon:
CREATE DATABASE spamassassin; GRANT INSERT,UPDATE,DELETE,SELECT ON spamassassin.* TO 'spamassassin'@'%' IDENTIFIED BY 'XXXXXXXX';
Configurer /etc/spamassassin/sql.cf avec les users mysql:
user_scores_dsn DBI:mysql:spamassassin:localhost user_scores_sql_username spamassassin user_scores_sql_password XXXXXXXXXXXX auto_whitelist_factory Mail::SpamAssassin::SQLBasedAddrList user_awl_dsn DBI:mysql:spamassassin:localhost user_awl_sql_username spamassassin user_awl_sql_password XXXXXXXXXXXX user_awl_sql_table awl bayes_store_module Mail::SpamAssassin::BayesStore::SQL bayes_sql_dsn DBI:mysql:spamassassin:localhost bayes_sql_username spamassassin bayes_sql_password XXXXXXXXXXXX
Protéger le fichier de config:
chmod 640 sql.cf chgrp debian-spamd sql.cf
Ensuite, charger les schémas de base de données:
for i in awl bayes txrep userpref ; do mysql spamassassin < <(cat /usr/share/doc/spamassassin/sql/$i_mysql.sql | sed 's/TYPE=MyISAM//') done
Le sed est nécessaire sinon on a l'erreur suivante sur un install par défaut de MySQL dans Debian Wheezy:
ERROR 1064 (42000) at line 1: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'TYPE=MyISAM' at line 8
application des nouvelles configurations
redémarrage du service spamassassin
service spamassassin restart
Différents tests
Tester si le daemon fonctionne bien:
spamc -u test@localhost --dest=localhost < /usr/share/doc/spamassassin/examples/sample-spam.txt gunzip -c /usr/share/doc/spamassassin/examples/sample-nonspam.txt.gz | spamc -u test@locahost --dest=localhost
--dest=localhost est nécessaire, sinon on obtient
spamd: unauthorized connection from ::1 [::1]:48290 to port 783, fd 6 at /usr/sbin/spamd line 1580.
Essayer d'envoyer un courriel pour confirmer qu'il est bien identifié comme "spam"
On veut le sujet suivant:
XJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-UBE-TEST-EMAIL*C.34X
spamassassin devrait l'identifier comme spam.
2 autres méthodes pour faire des tests ici: https://easyengine.io/tutorials/mail/server/testing/spam/
Déployer avec postfix milter
Installer spamass-milter:
apt install spamass-milter
Ajouter le path de connection à smtpd_milters dans main.cf:
non_smtpd_milters = $non_smtpd_milters smtpd_milters = unix:/var/spool/postfix/spamass/spamass-milter.sock # inet:127.0.0.1:8891 # (dkim)
Pourquoi? Le spamassassin-milter est en pont entre postfix et spamd qui nous permet d'utiliser des checks milter (milter_header_checks) pour traiter les messages d'un manière plus costaud, eg. rejeter tout ce qui à un score >= XXX etc.
N.B Si tout l'entrainement sur le serveur est sur un utilisateur, eg debian-spamd, ajouter
OPTIONS="${OPTIONS} -- -u debian-spamd"à la fin de /etc/default/spamass-milter
N.B. C'est pas tant important d'avoir spamassassin dans le non_smtpd_milters. Voir les docs de postfix sure où les milters sont appliqués
Déployer avec postfix "content filter"
On ajout les lignes suivantes au fichier /etc/postfix/master.cf
--- a/master.cf
+++ b/master.cf
@@ -14,6 +14,10 @@
# (yes) (yes) (yes) (never) (100)
# ==========================================================================
smtp inet n - - - - smtpd
+ -o content_filter=spamassassin
+spamassassin unix - n n - - pipe
+ user=debian-spamd argv=/usr/bin/spamc --dest=localhost -f -e
+ /usr/sbin/sendmail -oi -f ${sender} ${recipient}
submission inet n - - - - smtpdredémarrage du service postfix
service postfix restart
Vérfier les enveloppes des courriels, vous avez aller maintenant avoir des nouvelles lignes
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on hostname X-Spam-Level: ** X-Spam-Status:
voir les logs...
# tail -f /var/log/syslog | grep spamd
Si postfix utilise DNS(DNSBL) de spamhaus ...
Spamassassin par défaut fait aussi une vérification sur le service de spamhaus, on ne veut pas dupliquer la tâche, on va le désactiver dans spamassassin et laisser postfix faire le boulot. Ça semble relatif au désir des clients. On note que spamhaus fait un premier ménage pour pas qu'on traite trop de spam par spamassassin, ensuite spamassassin fait des vérifications avec d'autres services de DNSBL donc, on veut garder la vérification active dans spamassassin.
Déployer avec .forward/procmail
spamc peut être intégré à un fichier .forward, à procmail
Exemple pour RT:
# file managed through puppet
:0fw: spamassassin.lock
* < 300000
| /usr/bin/spamc -u rt@rt.example.net
#if the spam trigger is fired send to spam queue
:0w
* ^(X-Spam-Status: Yes|X-DSPAM-Result: Spam)
{
QUEUE=Spam
}Exemple procmail:
:0 * ^X-Spam-Status: Yes .junk/
À considérer
# loadplugin Mail::SpamAssassin::Plugin::Rule2XSBody
... dans /etc/spamassassin/v320.pre
pour les pourriels en d'autres langues...
https://spamassassin.apache.org/full/3.2.x/doc/Mail_SpamAssassin_Conf.html
diff --git a/spamassassin/v310.pre b/spamassassin/v310.pre index 56e5b2f..55f3e83 100644 --- a/spamassassin/v310.pre +++ b/spamassassin/v310.pre @@ -50,7 +50,7 @@ loadplugin Mail::SpamAssassin::Plugin::AutoLearnThreshold # TextCat - language guesser # -#loadplugin Mail::SpamAssassin::Plugin::TextCat +loadplugin Mail::SpamAssassin::Plugin::TextCat
diff --git a/spamassassin/local.cf b/spamassassin/local.cf index 7c3407a..9ecd86c 100644 --- a/spamassassin/local.cf +++ b/spamassassin/local.cf @@ -7,6 +7,15 @@ # ########################################################################### +# conf comme demandé ici : https://redmine.koumbit.net/issues/12087 +# language config, requires Mail::SpamAssassin::Plugin::TextCat +ok_languages de en es fr pt + +# demandé https://redmine.koumbit.net/issues/12087 +# comme documenté +# https://spamassassin.apache.org/full/3.2.x/doc/Mail_SpamAssassin_Conf.html +ok_locales en +
activer le suivi de spamassassin par munin
On veut pouvoir la quantité de détection de spam par spamassassin. Rien de mieux qu'un plugin munin, le plugin est disponible dans le package munin-plugins-extra
Si on veut l'ajouter en utilisant puppet,
munin::plugin { 'spamstats':
ensure => present,
config => "group adm\n";
}
Déplacer les spams dans le répertoire Junk (avec Dovecot et sieve)
Dans la version AlternC 3.2.10, on a Dovecot et le filtrage des courriels avec Sieve. Nous allons utiliser une règle global dans sieve pour déplacer un courriel détecté dans le répertoire Junk de l'utilisateur. Vérifier si le répertoire Junk ou INBOX.Junk existe.
mkdir /etc/dovecot/sieve/
cat >> /etc/dovecot/sieve/default.sieve << EOF
require "fileinto";
if header :contains "X-Spam-Flag" "YES" {
fileinto "Junk";
}
EOFOn compile le filtre
sievec /etc/dovecot/sieve/default.sieve
On ajuste les permissions:
chown -R vmail:vmail /etc/dovecot/sieve/
Ajouter la configuration à dovecot pour activer les filtres globaux et qui doivent être utilisé avant les filtres personalisés.
Au fichier suivant: /etc/dovecot/conf.d/95_alternc.conf
plugin {
....
sieve_global_path = /etc/dovecot/sieve/default.sieve
sieve_global_dir = /etc/dovecot/sieve/
sieve_before = /etc/dovecot/sieve/
}
débug
Si vous activé le débug dans dovecot, vous allez voir une ligne comme ça dans mail.log, si ça fonctionne bien.
Debug: sieve: script binary /etc/dovecot/sieve/default.svbin successfully loaded
Ensuite, sans le débug en mode normal, vous allez retrouver une ligne comme ça quand le filtre s'applique.
stored mail into mailbox 'INBOX.Junk'
reférence: http://wiki1.dovecot.org/LDA/Sieve/Dovecot
Activer les plugins dans roundcube
Activer le plugin "markasjunk" dans la conf de roundcube, dans le fichier /etc/roundcube/config.inc.php
:$rcmail_config['plugins'] = array("managesieve","password","markasjunk");markasjunk2 semble vraiment intéressant mais il n'est pas distribué dans jessie, on va attendre strech.
Postfix simple content filter configuration
Following the postfix integration guide, I have performed the following surgery:
--- master.cf.orig Tue Apr 18 16:27:30 2006
+++ master.cf Tue Apr 18 16:27:31 2006
@@ -6,7 +6,7 @@
# service type private unpriv chroot wakeup maxproc command + args
# (yes) (yes) (yes) (never) (100)
# ==========================================================================
-smtp inet n - n - - smtpd
+smtp inet n - n - - smtpd -o content_filter=spamassassin
#submission inet n - n - - smtpd
# -o smtpd_etrn_restrictions=reject
# -o smtpd_client_restrictions=permit_sasl_authenticated,reject
@@ -77,3 +77,5 @@
flags=F user=ftn argv=/usr/lib/ifmail/ifmail -r $nexthop ($recipient)
bsmtp unix - n n - - pipe
flags=Fq. user=foo argv=/usr/local/sbin/bsmtp -f $sender $nexthop $recipient
+spamassassin
+ unix - n n - - pipe user=postfix argv=/usr/local/bin/spamc -u ${recipient} -e /usr/sbin/sendmail -oi -f ${sender} ${recipient}This way postfix pipes all incoming mails through spamc.
alternc.fqccl.org specifics
/etc/procmailrc
UMASK=007
RECIPIENT=$1
USR=`echo ${RECIPIENT} | grep -o '^[^@]*'`
LETTER=`echo ${USR} | head -c 1`
MAILDIR=/var/alternc/mail/$LETTER/$USR/Maildir
DEFAULT=${MAILDIR}/
:0
* ^X-Spam-Status: Yes
${MAILDIR}/.spam/
master.cf
'procmail' and 'spamassassin' transports added in /etc/postfix/master.cf, filter added as per above (user=spamassassin) :
procmail unix - n n - - pipe
flags=Rhu user=www-data argv=/usr/bin/procmail -a ${recipient} -p /etc/procmailrc
spamassassin
unix - n n - - pipe user=spamassassin argv=/usr/bin/spamc -e /usr/sbin/sendmail -oi -f ${sender} ${recipient}
main.cf
default transport changed from 'local' to 'procmail' in /etc/postfix/main.cf:
local_transport = procmail
We also need to tell postfix not to deliver too many emails to procmail at the same time, since it only knows how to deal with one at a time:
procmail_destination_recipient_limit = 1
/usr/local/sbin/expire_spam
find /var/alternc/mail -type f -mtime +7 -iregex '^.*Maildir/\.spam/.*' -exec rm -v {} \;
/etc/crontab
11 3 * * * www-data /usr/local/sbin/expire_spam
Advanced content filter with Amavis
The above works well for small sites, but doesn't degenerate gracefully, as it will bounce mail when out of memory or resources, for example. The solution is to setup a content filter using Amavis, as described in the FILTER_README.
Quick recall, the general principle is to have a parallel SMTP server (Amavis) that is designed to accept mail to be scanned, and send it back (or not) into the mailserver. Here is a general picture to help us remember what port numbers we're talking about here:
+----------+ +-------+ +--------------+ +-------------+
| smtpd:25 | -> | queue | -> | amavis:10024 | -> | smtpd:10025 |
+----------+ +-------+ +--------------+ +-------------+
^ |
| |
+------------------------------------+smtpd:25 is the regular SMTP server that takes incoming mail. queue is the regular postfix queue. amavis:10024 is the amavis mailserver, running on port 10024, and smtpd:10025 is a special postfix SMTP server that takes filtered mail and reinjects it in the queue properly (so that it doesn't get filtered again, for example).
Amavis setup
(aptitude install amavisd-new...)
We setup amavis to listen on eth1 for 2 reasons: (1) it doesn't seem to want to do it on 127.0.0.1, probably because of the vserver, and (2) because it can be used by other servers on the internal network later on.
The code below is all in /etc/amavis/amavisd.conf.
$inet_socket_bind = '192.168.0.104'; @inet_acl = qw( 127.0.0.1 192.168.0.104 );
$forward_method = 'smtp:192.168.0.104:10025'; # where to forward checked mail $notify_method = $forward_method; # where to submit notifications
Don't modify subject if not scanned:
$undecipherable_subject_tag = undef;
Spamassassin setup in Amavis
Because some settings are overridden by amavis (??), they need to be done in /etc/amavis/amavisd.conf:
$sa_tag_level_deflt = undef; # always tag messages
$sa_tag2_level_deflt = 5; # add 'spam detected' headers at that level
$sa_kill_level_deflt = $sa_tag2_level_deflt; # triggers spam evasive actions
# at or above that level: bounce/reject/drop,
# quarantine, and adding mail address extension
$sa_dsn_cutoff_level = 10; # spam level beyond which a DSN is not sent,
# effectively turning D_BOUNCE into D_DISCARD;
# undef disables this feature and is a default# a quick reference: # tag_level controls adding the X-Spam-Status and X-Spam-Level headers, # tag2_level controls adding 'X-Spam-Flag: YES', and editing Subject, # kill_level controls 'evasive actions' (reject, quarantine, extensions); # it only makes sense to maintain the relationship: # tag_level <= tag2_level <= kill_level < $sa_dsn_cutoff_level
Since we treat evil spam elsewhere (in procmail), we let it pass through amavis:
$final_virus_destiny = D_DISCARD; # (defaults to D_BOUNCE) $final_banned_destiny = D_BOUNCE; # (defaults to D_BOUNCE) $final_spam_destiny = D_PASS; # (defaults to D_REJECT) $final_bad_header_destiny = D_PASS; # (defaults to D_PASS), D_BOUNCE suggested
... and don't quarantine it:
$spam_quarantine_to = undef;
In /etc/spamassassin/local.cf, we configure more stuff, in particular:
# Save spam messages as a message/rfc822 MIME attachment instead of # modifying the original message (0: off, 2: use text/plain instead) report_safe 1
chroot
Voir README.chroot.txt pour l'implantation, pas l'air si simple...
Accepting mail from the content filter
In /etc/postfix/master.cf:
192.168.0.104:10025 inet n - n - 10 smtpd
-o content_filter=
-o receive_override_options=no_unknown_recipient_checks,no_header_body_checks
-o smtpd_helo_restrictions=
-o smtpd_client_restrictions=
-o smtpd_sender_restrictions=
-o smtpd_recipient_restrictions=permit_mynetworks,reject
-o mynetworks=192.168.0.0/24
-o smtpd_authorized_xforward_hosts=192.168.0.104don't forget a postfix reload.
First tests
You can right now test if amavis can receive mail and send it back into postfix:
swaks -t anarcat@koumbit.org -s 192.168.0.104 -p 10024
Basically, using this, we're able to test the filtering capacities without hooking them into the main queue and SMTP server, by talking directly to Amavis.
Antivirus (ClamAv) setup
We start by leaving Clamav disabled:
@av_scanners = (); @av_scanners_backup =();
This should put after the default definitions so that no AV is defined and therefore tried.
Spamassassin setup
aptitude install spamassassin. well, not exactly, we install the version from backports.org using the GUI. we use release 3.1 because of the following features:
- added SPF-based whitelisting.
new plugins: DomainKeys (off by default)
We remove spamc because we do not use it.
To activate spamassassin in amavis, we comment out the line:
#@bypass_spam_checks_acl = qw( . ); # No default dependency on spamassassin
in amavisd.conf and of course restart amavis.
Final deployment
check if it's not better to use amavis' quarantine instead of procmail for isolating spam...
These are the steps currently planned in the deployment:
Notify users (day 0)
Activate filtering in postfix (day 0)
Hook in monitors (day 0)
- Test performance (day 0)
Activate procmail filters & renotify users (day 7)
- Consider Clamav
- Consider learning capabilities
Notify users
Users will have to be notified of the new service and its impacts. Important facts:
- mail will initially not be filtered, only tagged, for 1 week
- when filtered away, it will be stored in a junk subfolder, available through IMAP or webmail
- one month (28 days) old email in the junk folder will be deleted each night
spam-learning and virus scanning might be deployed in the near future
/FirstNotification was sent today. -- TheAnarcat 2006-08-14 10:51:57
Activate filtering in postfix
In master.cf:
scan unix - - n - 2 smtp
-o smtp_send_xforward_command=yes
-o disable_mime_output_conversion=yesThe "2" there is the process limit, in sync with amavis.
In main.cf:
content_filter = scan:192.168.0.104:10024 receive_override_options = no_address_mappings
Hook in monitors
See MuninService/SpamassassinPlugin.
Test performance
Look for differences in deliveries delays. Compare those two graphs:
Activate procmail filters & renotify users
Consider the alternc-procmail package to allow people to create their own rules, while we're at it.
This consists mostly in duplicating the procmail setup for alternc.fqccl.org described above. This includes setting up the cronjob, but for 28-days checks.
Also consider the capabilities of postfix and amavis regarding address extensions: amavis could just forward spam email to foo+junk@example.com instead of letting procmail refilter the mail.
Consider Clamav
Test and deploy an antivirus.
Consider learning capabilities
See if it's possible to have amavis take care of storing preferences per-recipient instead of per-UNIX user, as is usually the case with spamc/spamd.
performance analysis
- amavis is said to be at least as fast as spamc/spamd because it loads spamassassin internally
- the "layered" approach allows for easily offloading the service to another host
check if spam checks are done for non-existent emails - check of compliance with requesites above
With spampd
Seems like a simple enough configuration:
apt-get install spampd vi /etc/postfix/master.cf postfix reload
Modifications required to master.cf:
smtp inet n - - - - smtpd
-o content_filter=smtp:127.0.0.1:10025
-o myhostname=mx.anarcat.ath.cx
localhost:10026 inet n - n - 10 smtpd
-o content_filter=
-o myhostname=delivery.anarcat.ath.cx... that's all! spampd talks with spamassassin and uses the normal configuration.
Above config out of date, used by anarcat on his home setup, see http://anarc.at/hardware/server/marcos/configuration/?updated#index1h4 for details.
Simple lethe / FreeBSD install notes
That was a pain. After the perl 5.8 upgrade, I had to reinstall it, but then it was dependent upon p5-Net, itself on p5-Authen-SASL and then p5-GSSAPI which didn't compile, probably because I didn't have kerberos compiled on the machine. I commented the dependency in p5-Authen-SASL and all was merry.
Démarrer le filtre Bayesien
Voir: SpamAssassinMaintenance#Apprentissage_Bayesien
Original specifications
required for FQCCL
don't filter for non-existent accounts
junk must go in a junk subfolder
- virus detection must be included with Amavisd, which should probably be taking care of talking to spamassassin
junk mail is deleted after 4 weeks (7 days for fqccl) (ie. >= 4 weeks old junk mail is deleted each night)
- bayesian filtering and spamassassin preferences either
per email addresses (will require hacking because spamc -u requires a real username)
- per alternc account (will also require hacking)
- learning can be done by putting spam in a spam subfolder and ham in a ham subfolder
MuninService/SpamassassinPlugin setup to graph spam detection
don't affect too much performance
Voir PropositionSpamAssassinAmavis.
References
http://wiki.apache.org/spamassassin/UsingSiteWide, in particular:
postfix FILTER_README
using amavisd-new with spamassassin and clamav in Debian Squeeze shows how config is organized under Squeeze.