More servers installed to deal with spam load

We’ve now setup and installed two new servers to deal with the increased mail delivery load that’s been occurring the last two days. These new servers are powerful dual core Xeon 5130 processor based servers, and so should be able to handle the processing load. We’ve also lowered the queue warning level, so we’ll be paged earlier if there does appear to be any excess email building up so we can deal with it quicker.

With the new servers, we’ve also taken some time to update our SpamAssassin installation to the latest version, and also install the FuzzyOcr plugin to try and deal with the large number of stock “pump-and-dump” image spams that are plaguing Internet email users at the moment.

Posted in Technical. Comments Off

Delayed inbound mail again

It seems all the measures we put in place yesterday weren’t enough. We’ve just been paged by our servers that an inbound delayed mail queue has built up again. We’re seeing what we can do bring it back down.

Update (2:30pm EST): The queues are going down. We’re getting NYI to install 2 new servers ASAP that we previously were going to use for IMAP expansion and try and bring those up to help with the processing.

Update (3:05pm EST): Over half the mail from the queues is now delivered. Any new mail arriving should be delivered within about 1-2 minutes, old mail is being delivered as the queue can be cleared.

Update (4:05pm EST): Almost all of the queues have cleared now and all new mail should be delivered immediately.

Posted in Technical. Comments Off

Email delays yesterday

It seems today the spam attack we’ve been experiencing for the last week intensified even more, and caused email to backup on some of our servers causing some email to be delayed up to several hours. We’ve analysed what happened and believe we now have procedures in place to stop this occurring tomorrow and in the future.

Posted in Technical. Comments Off

Massive increase in email connections to our servers

About 4 days ago or so, something went crazy with the spam zombie machines out there. Previously the spam sending software spammers were using was acting like reasonably well behaved email sending software. It would connect to us, trying to send it’s spam, then disconnect just like any other email sending system on the Internet. They’d do that every now and then, maybe an hour or two between attempts. Still, with 100,000’s of machines, that’s millions of attempts a day to send spam.

Now however, the zombie machines and software have just gone insane and are connecting over and over every few minutes, but mostly doing nothing during the connection. While that in theory might seem fine since they do nothing, it’s not. When you have 200,000+ machines connecting to you every few minutes, even if they do nothing you still have the connections to deal with, the RBL DNS lookups, the rate limiting lookups, etc.

The result was a significant jump in load on the incoming servers, significantly above the load jump we’ve seen over the last couple of months even.

To combat this, we’ve had to invoke some old code from a previous “bombing” attempt we had a while back. This code continuously scans the logs looking for particular aberrant behavior and then put those IPs on a special “early” block list which means as soon as the machine connects, it’s sent a response of:

454 Service temporarily unavailable; Client host [x.y.z.a] blocked using internal list; Access denied

And disconnected. Over the course of a couple of hours and days (as infected computers out there were turned on and off), we’ve built up a list of over 200,000 IPs that are now being “early blocked” like this. To give you an idea of how big the surge is, almost 3/4’s of all connections are now being “early blocked” by this list. That means incoming connections have probably almost tripled in in the last 4 days.

This is also something we can just confirm by the size of our log files. Normally our email processing files are rotated each day, but we’re now having to rotate them multiple times a day because they’re reaching their limit of 2 gigabytes in size!

Our only current worry is that somehow we’ve blocked some other services incorrectly. We’ve had one report from a user who’s scanner has been blocked (it’s an internet enabled scanner that you can setup to email you when you scan something, unfortunately it seems to be designed for LAN networks, and polls the SMTP server you’ve setup every 60 seconds to see if it’s alive, much like the spam zombies are *sigh*).

Some more information about the current spam wave that’s going on is at Extreme Tech.

Update: It seems some badly run sites were being blocked. Some sites with incorrect DNS setup were being identified as “dialup/dsl” machines. Some other sites seemed to be doing the same signature of the spam zombies, namely “connect, do nothing, disconnect”. Some other sites were sending rapidly to many unknown recipients, also a sign of a spam zombie trying to enumerate usernames. We’ve tightened up the blocking criteria some more, removed a number of existing blocks, and put some common hosts on an IP whitelist so they’re not blocked again in the future.

Posted in Technical. Comments Off

Use Ctrl-Shift-Enter to Send message on the Compose screen in Firefox

For several years, FastMail has allowed you to use Ctrl-Enter on the Compose screen as a shortcut to send a message. Unfortunately Firefox 1.5 and 2.0 both have a bug that means this doesn’t work as expected, instead showing the “Download manager” dialog as well as attempting to send the message. This bug in Firefox is documented and being tracked at the bugzilla site here.

Someone just recently noted that using Ctrl-Shift-Enter doesn’t trigger the bug, but because it still does have the Ctrl key pressed, it triggers the correct JavaScript code to send the message. So for Firefox 1.5 and 2.0 users, this is a good work around until this bug is fixed, just use Ctrl-Shift-Enter instead.

Posted in Technical. Comments Off

A number of small updates

A number of small updates have just been rolled out.

  1. When forwarding a number of messages as attachments, those messages would have been marked as read if they were unread. This no longer occurs, message stay unread
  2. When sending an email, on the next screen you’re presented with a list of addresses the email was sent to. Additionally now you’re also told what folder the sent message was saved into if any was specified explicitly or via the personality you used
  3. If you try and setup a Pop Link to a Hotmail account that has DAV access disabled, you’re now given a more informative message
  4. On Windows Mobile based PDAs using Internet Explorer should now be more reliable. In particular certain web operations (most commonly the Reply or Forward buttons on the message read screen) would cause system error messages to be returned. These should be fixed.
Posted in Technical. Comments Off

Domain split

We’ve now rolled out the ‘domain split’.

This means that you can use a username that someone else is using, as long as you choose a different domain.  So now, a lot of previously unavailable username – domain combinations can be used for aliases or new users.

Posted in News. Comments Off
Follow

Get every new post delivered to your Inbox.

Join 5,254 other followers