Improved spam filtering: avoiding email forwarding services

In a recent blog post, I mentioned that one of the ways to improve spam filtering is to avoid using a forwarding service, and especially if you have your own domain, to point the MX records for your domain directly to our servers (Enhanced or family/business accounts only).

The reason for this is that FastMail does a lot of work at the SMTP stage (when email is transferred from an external system to FastMail) to try and identify spam bots and block them while letting legitimate through. Unfortunately many of these processes can only work when the MX records for your domain point directly to our servers. If you point the MX records to another system, and forward from that system to FastMail, we can’t do the spam bot detection, and so more spam will get through.

To give you an idea of how effective this is, one of our users – the author, Simon Cann (http://simoncann.com/) – recently changed over the MX records for his Noise Sculpture domain (http://noisesculpture.com) to point to us instead of using a forwarding service. After doing so, he had the following to say:

As you suggested, I pointed the MX records for the Noise Sculpture domain at Fastmail and have been blown away by the results.  I’m now getting maybe three or four pieces of spam in a day and the few pieces of spam that are getting through are being correctly identified as spam and put in the Junk Mail folder.

Simon went on to explain why he hadn’t made the change sooner.

I came to Fastmail a year ago following an unexpected and highly inconvenient melt down of my web host who was also by default my email host for my simoncann.com domain.  This was embarrassing and potentially lost me business (I was in the middle of pitching a book proposal to a group of literary agents).

On signing up with Fastmail I immediately pointed the simoncann.com MX record at Fastmail.  My other website for my music-related books (Noise Sculpture) was hosted by a different web host and so I forwarded all of my email from that domain to Fastmail.  I didn’t see a need to change the MX record–the web host was working well (still is) and I couldn’t be bothered to start hacking the DNS records.

Long story short, as the simoncann.com and noisesculpture.com emails were sent to a single Fastmail account, I hadn’t realized how much simoncann.com spam was getting killed by Fastmail.  However, now that I have changed the MX record for noisesculpture.com, I can see just how much spam was getting through from that domain because I was using forwarding rather than pointing the MX records.

This change has stunned me (in a very good way).  I’m so impressed with the change (allied with your excellent service) that I’ve just renewed my Fastmail subscription to cover the next few years.

For the technically interested, the main things we do at the SMTP delivery stage to detect these spam bots are:

  • XBL blocking. The XBL is a highly accurate block list that lists the IPs of machines with known trojans and proxies. Independent testing shows that the ZEN RBL (of which XBL is a part) has a high block rate, with basically no false positives
  • Address enumeration detection. Spammers are always trying to find new addresses to send their spam to, and they do that by enumerating email addresses at hosts. We have a system that will automatically detect this behaviour, and block the spam bot from discovering any additional information about whether an address is valid or not
  • Greylisting. Greylisting is a process designed to detect poorly written sending email servers, of which almost all spam bots are. While greylisting is very useful, it is sometimes frowned upon because poorly implemented versions would delay email delivery. The FastMail implementation uses a number of tweaks to ensure that legitimate email is delivered with no delay, while spam email is delayed or stopped from delivery
  • RFC violations. When a server is transferring email from one system to another, there’s a series of conventions they have to follow defined in the SMTP RFC 2821. Again, spam bots are often poorly implemented, or try and use tricks to work around this conventions. By detecting some of the common violations, more spam bots can be detected and blocked
  • Custom spambot detection. Through our large and long term experience with running an email service, we’ve been able to closely analyse how spam bots try to deliver email to our servers. By using this data, we’re able to pre-emptively detect spam bots early only after they connect, and block them from delivering their spam email at all
Posted in News, Technical. Comments Off
Follow

Get every new post delivered to your Inbox.

Join 5,162 other followers

%d bloggers like this: