I hate spam
December 30th, 2009We provide mail services for our customers and we want to keep their inboxes free of spam. To fight spam we use a couple of techniques I would like to tell you about. I’ll show some excerpts of Postfix’s main.cf and related files.
Common Postfix techniques
Postfix offers some techniques to make sure that the remote client behaves nicely.
smtpd_recipient_restrictions = ... reject_non_fqdn_recipient, reject_non_fqdn_sender, reject_unknown_sender_domain, reject_unauth_destination, #reject_invalid_helo_hostname, #reject_unknown_helo_hostname, reject_non_fqdn_helo_hostname, reject_unverified_recipient, ... smtpd_helo_required = yes disable_vrfy_command = yes strict_rfc821_envelopes = yes body_checks = regexp:/opt/csw/etc/postfix/maps/body_checks header_checks = regexp:/opt/csw/etc/postfix/maps/header_checks mime_header_checks = regexp:/opt/csw/etc/postfix/maps/mime_header_checks unknown_address_reject_code = 550
reject_non_fqdn_recipient and reject_non_fqdn_sender make sure that the sender and recipient addresses are real mail addresses like user@example.com. reject_unknown_sender_domain rejects mail when the domain of the sender mail address has no DNS A or MX record and unknown_address_reject_code returns a permanent 550 error when this happens.
smtpd_helo_required requires a HELO or EHLO at the beginning of an SMTP session. We wan’t the HELO or EHLO hostname to be a fqdn (reject_no
n_fqdn_helo_hostname). reject_invalid_helo_hostname and reject_unknown_helo_hostname gave too much trouble with broken but valid clients, mostly M$ Exchange servers at offices.
With reject_unauth_destination we make sure we only accept mail when we are the final destination or when we act as a relay for the final destination. If we are a relay we want to make sure that the recipient is a valid recipient reject_unverified_recipient, if we forget to check this we could be sending a lot of backscatter.
The *_checks do some simple regex checks on the content of the message. I copied these checks from Jeffrey Posluns’ Postfix Guides.
The strict_rfc821_envelopes parameter controls how tolerant Postfix is with respect to addresses given in MAIL FROM or RCPT TO commands. Unfortunately, the widely-used Sendmail program tolerates lots of non-standard behavior, so a lot of software expects to get away with it. Being strict to the RFC not only stops unwanted mail, it also blocks legitimate mail from poorly-written mail applications but we never had any issues with that.
Policy server
We used to list DNSBLs directly in main.cf but that caused too much troubles. Sometimes valid clients were listed on a blacklist and so we blocked their mail. To prevent being dependent on the judgment of a single blacklist we moved to a policy server. A policy server computes a score and then decides to accept or to reject a mail. We use policyd-weight as a policy server for Postfix. Besides the blacklist checks policyd-weight also does some other checks like is the sender on a dialup network, is the sending host listed as a DNS A or MX record for the domain of the sender mail address and is the HELO or EHLO hostname the same as the hostname of the client.
@client_ip_eq_helo_score = (0.5, -1.25 );
@helo_score = (0.5, -2 );
@helo_from_mx_eq_ip_score = (0.5, -3.1 );
@helo_numeric_score = (2.5, 0 );
@from_match_regex_verified_helo = (1, -2 );
@from_match_regex_unverified_helo = (1.6, -1.5 );
@from_match_regex_failed_helo = (2.5, 0 );
@helo_seems_dialup = (1.5, 0 );
@failed_helo_seems_dialup = (2, 0 );
@helo_ip_in_client_subnet = (0, -1.2 );
@helo_ip_in_cl16_subnet = (0, -0.41 );
@client_seems_dialup_score = (3.75, 0 );
@from_multiparted = (1.09, 0 );
@from_anon = (1.17, 0 );
@bogus_mx_score = (1, 0 );
@random_sender_score = (0.25, 0 );
@rhsbl_penalty_score = (3.1, 0 );
@enforce_dyndns_score = (3, 0 );
# HOST, HIT SCORE, MISS SCORE, LOG NAME
@dnsbl_score = (
'pbl.spamhaus.org', 3.25, -2, 'DYN_PBL_SPAMHAUS',
'sbl-xbl.spamhaus.org', 4.35, -3.5, 'SBL_XBL_SPAMHAUS',
'bl.spamcop.net', 3.75, -2.5, 'SPAMCOP',
'dnsbl.njabl.org', 4.25, -2.5, 'BL_NJABL',
'list.dsbl.org', 4.35, -1, 'DSBL_ORG',
'ix.dnsbl.manitu.net', 4.35, 0, 'IX_MANITU',
'dnsbl.sorbs.net', 3.25, 0, 'SORBS',
'b.barracudacentral.org', 3.25, -1, 'BARRACUDA_CENTRAL',
'spam.abuse.ch', 0.5, 0, 'ABUSE_CH',
'dnsbl-1.uceprotect.net', 2.25, 0, 'DNSBL1_UCEPROTECT',
'dnsbl-2.uceprotect.net', 1.25, 0, 'DNSBL2_UCEPROTECT',
'dnsbl-3.uceprotect.net', 1.00, 0, 'DNSBL3_UCEPROTECT',
);
@rhsbl_score = (
'multi.surbl.org', 4, 0, 'SURBL',
'rhsbl.ahbl.org', 4, 0, 'AHBL',
'dsn.rfc-ignorant.org', 3.5, 0, 'DSN_RFCI',
'postmaster.rfc-ignorant.org', 0.1, 0, 'PM_RFCI',
'abuse.rfc-ignorant.org', 0.1, 0, 'ABUSE_RFCI'
);
After a lot of tweaking, which still is an ongoing process, we use the above settings for policyd-weight.
To enable the policy server we added this line to main.cf
smtpd_recipient_restrictions = ... check_policy_service inet:127.0.0.1:12525, ...
Greylisting and whitelisting
We use greylisting as a cheap (it doesn’t cost a lot of cpu power) technique to stop spammers. When a client sends a mail for the first time to a recipient Postfix returns a 450 error. This is a temporary error and the client must try again later. Some spammers don’t try again after a 450 so it’s a effective way to stop some spam. The disadvantage of this technique is that it can cause some delay in mail delivery. This delay will be just a couple of minutes most of the time.
This graph shows the number of greylisted messages and the number of delayed messages. A quick calculation show that 39% of the messages that are greylisted are spam.
Greylisting and whitelisting
We use tumgreyspf as a greylist and SPF (more on SPF later) policy server. It’s configuration is straight forward.
The prevent greylisting known valid clients we use a whitelist. When the client is on the whitelist we skip greylisting.
smtpd_recipient_restrictions = ... check_client_access cidr:/opt/csw/etc/postfix/maps/dnswl_header, check_client_access cidr:/opt/csw/etc/postfix/maps/dnswl_permit, check_policy_service inet:127.0.0.1:10023, ...
To update these maps we use this script:
/opt/csw/bin/rsync \ --times rsync1.dnswl.org::dnswl/postfix-dnswl-permit \ /opt/csw/etc/postfix/maps/ /opt/csw/bin/rsync \ --times rsync1.dnswl.org::dnswl/postfix-dnswl-header \ /opt/csw/etc/postfix/maps/ cat /opt/csw/etc/postfix/maps/postfix-dnswl-header | \ sed "s/X-REPLACEME/X-YoungGuns-WhiteList/" > \ /opt/csw/etc/postfix/maps/dnswl_header cp /opt/csw/etc/postfix/maps/postfix-dnswl-permit \ /opt/csw/etc/postfix/maps/dnswl_permit
SPF check
As the name may suggest tumgreyspf also checks SPF records for the sender mail address. If the check fails the mail is rejected. More on SPF can be found on www.openspf.org and a tool to check SPF records can be found on www.kitterman.com/spf/validate.html.
Don’t route or peer and bogon networks
DROP is an advisory drop all traffic list, consisting of stolen or hijacked netblocks and netblocks controlled entirely by professional spammers. The list is maintained by Spamhaus.
Bogon is an informal name for an IP packet on the public Internet that claims to be from an area of the reserved IP address space, but not yet allocated or delegated by the IANA or a delegated RIR. The areas of unallocated address space are called bogon space. When the DNS MX record of the sender point to an bogon address we reject their mail.
smtpd_recipient_restrictions = ... check_sender_mx_access cidr:/opt/csw/etc/postfix/maps/bogon_networks, check_client_access cidr:/opt/csw/etc/postfix/maps/drop, check_sender_mx_access cidr:/opt/csw/etc/postfix/maps/drop, check_sender_ns_access cidr:/opt/csw/etc/postfix/maps/drop, ...
To update these maps we use this script:
wget -q -nd --output-document=- \
http://www.spamhaus.org/drop/drop.lasso | \
awk '/; SBL/ {printf("%s\tREJECT %s\n", $1, $3)}' > \
/opt/csw/etc/postfix/maps/drop
wget -q -nd --output-document=- \
http://www.cymru.com/Documents/bogon-bn-agg.txt | \
awk '{printf("%s\tREJECT IP address of MX host is in bogon netspace\n", $0)}' \
> /opt/csw/etc/postfix/maps/bogon_networks
Fail2ban
Fail2ban is a daemon that watches log files and bans IP addresses that make too many failures (like failed login attempts). It updates firewall rules to ban the IP address. We use it to reject an IP address when it generates too many 550 errors in the Postfix log or sends a spam message as detected by SpamAssassin. The banning is for a short period and it stops hammering spammers. Repeated short bans will lead to a longer ban of the IP address.
This graph shows the number of banned hosts during one month. You can clearly see a peak of banned hosts add the end of week 49. These connections where all banned with a simple firewall rule.
Blocked and banned hosts
Virus and SpamAssassin
Spam and viruses are closely related. A lot of viruses are installed on computers to send spam from that computer. We don’t want our customers to send spam which they are unaware of. And of course we don’t want our customer’s computer to be infected with any virus. We have a rule in main.cf to reject known clients which are sending viruses.
smtpd_recipient_restrictions = ... reject_rbl_client virbl.dnsbl.bit.nl, ... content_filter = amavisfeed:127.0.0.1:10024
We use a the improved amavisd-new to check the content of the message. Amavisd-new lets Clam AntiVirus and SpamAssassin filter the message. Messages with viruses are put in quarantine and not delivered to the user’s mailbox. Messages which are likely spam are delivered to a spam folder in the user’s mailbox.
The “Blocked and banned hosts” graph also showed the number of detected viruses in the last month, 1 virus! Most messages which contain viruses where already rejected beforehand because they where to spammy.
Conclusion
The combinations of these techniques result in an almost spam free Inbox for most users. We are continually monitoring the performance of all the techniques and make adjustments and improvements when necessary. This setup works for our situation and I can not guarantee that it will work for you. Please feel free to post comments or suggestions.