Push events, NAT TCP connection timeouts, and device sleep

This is a technical post. Regular FastMail users subscribed to receive email updates from the FastMail blog can just ignore this post.

When we released the new user interface last year, one of the improvements included was push updates when new emails arrived.

In theory, push events are conceptually quite easy to do. We open a connection from your web browser to the server (see this blog post for details), then when a new email arrives, we send a message down the open connection to let the browser know. It then fetches the details about the new email(s) and refreshes the display.

Unfortunately, in the real world, it’s not quite that easy. The biggest problem is that when a mailbox is mostly idle (no new mail arriving), the connection from the browser to the server will be idle. While this shouldn’t be a problem, it turns out it often is.

As we have noted before, some of our users are behind NAT gateways/stateful firewalls that have short state timeouts. If you leave a TCP connection idle for too long (variable from 2 to 30 minutes depending on the device), these start dropping any new packets on the connection.

In the case of a push connection from the server to the client, this is particularly bad. When a new email arrives, the server will try and send data to the client, and then be told the connection is dead at that point. That’s fine for the server, it can then clean up the connection. However, the client will never see any data from the server, and neither will the client ever know that the gateway/firewall has broken the connection. The client will think it is still connected to the server and has no way of knowing that the connection has actually been broken. This is purely a consequence of the way the TCP protocol works. The only way for the client to be able to tell the connection is broken is to send some data down the connection, and there are only 2 ways that can happen.

  1. If the client has enabled TCP keepalive on the socket. Currently only Chrome on Windows does this.
  2. If the client sent some data down the connection to the server. Unfortunately the eventsource specification doesn’t provide any way to do this; it basically assumes the underlying TCP connection is always reliable and only the server can send to the client.

One way to try and work around this issue is for the server to send regular “ping” events to the client, sufficiently often that the gateway/firewall knows the connection is still alive. This is relatively straightforward to do, but causes other problems.

If the ping events come too fast, it can cause some clients to never go into sleep mode. For instance, we used to send ping events every 60 seconds. It was noted that on an iPad if you left the FastMail webpage open in Safari and put the iPad down, the iPad itself would never actually go to sleep. The screen would stay on, draining the battery very quickly.

Because of that, we decided to go the other direction and disabled the ping events, but that ends up back at the other end of the scale where sometimes push just seems to randomly stop working.

As there’s no perfect solution to this problem, we’re now changing again to a new trade off.

  1. The server will send regular “ping” events to the client at 5 minute intervals. This should be enough for most gateways/firewalls to keep the connection open, but long enough apart to allow devices to go to sleep.
  2. If the client doesn’t see a ping event after 6 minutes, it assumes the connection has died, drops the existing connection and creates a new one. This should at least allow push events to work to some extent on connections with gateways/firewalls with low timeouts.

This change has now been rolled out everywhere. Based on initial testing, we think that this time we’ve got the balance between theory and reality right.

Posted in Technical. Comments Off

Google Authenticator now supported for two-factor authentication

FastMail has long supported various methods of two-factor authentication for additional account security, from generated one-time-passwords, to SMS, to Yubikey. Today we’ve added another method to our stable – the Google Authenticator method, otherwise known as Time-based One Time Passwords (RFC 6238). With this you can use your iOS, Android or almost any other mobile device as your second factor when authenticating, increasing the security on your account without requiring you to carry an additional object around with you.

A Google Authenticator alternative login can be configured in the Alternate Logins section of your account settings screen. If you’re using the official Google clients, then you can use its support for QR codes to make setup super-easy. You can however choose to use any number of other clients that support this authentication mechanism; all will work with our implementation.

Please refer to our Google Authenticator help page for more details.

Posted in Uncategorized. Comments Off

Reading pane available

Today we rolled out support for a longstanding feature request we’ve had here at FastMail: a reading pane in our web interface. Displaying the mailbox listing next to the selected conversation means you can go through your email without switching between two different screens, and you can see at a glance what other messages are in your mailbox whilst reading an email. This works particularly well in today’s age of widescreen computers and tablets, making good use of all that horizontal screen space.

You’ll find the option to choose a layout that shows the reading pane in the Settings, as part of the “Theme” group of settings. You’ll also find here an option to hide the sidebar, which is useful on smaller devices where you want to use the space for the reading pane instead. Note, when logging in on an iPad we automatically enable the reading pane and hide the sidebar to make optimal use of the space available.

The reading pane is not available in the classic interface.

Posted in News. Comments Off

Dropbox integration now available

“Cloud” storage is a big deal these days, and there’s all sorts of great ways to make your files available everywhere. FastMail has had an online file storage facility for years, but there’s other popular services out there too. So recently we started thinking about how we could let you use other file services from inside FastMail.

Dropbox is one of the most popular of these services, and we think they do a great job of making it easy to access your files everywhere, no matter what kind of device you use. That philosophy fits well with what we’re trying to do with FastMail, so it seemed like a natural fit to make the two services work together.

So today we’ve released our Dropbox integration. Now when you attempt to attach a file, you’ll be offered a new option: “Attach From Dropbox”. The first time you use this you’ll be asked to sign in to Dropbox and authorise FastMail, but after that you’ll be able to browse and attach files from Dropbox just as easily as you can from your FastMail file storage. Similarly, you can save attachments from your messages directly to Dropbox.

We think this is pretty great. Now you can do things like save a document to the Dropbox folder on your computer at home, then attach it to an email you write from a friend’s computer. you could save some photos you received in a message to your Dropbox, and then use the Dropbox app on your phone to access and share them, and have them ready for you when you get home without having to do anything else.

Note that at no time does this give Dropbox access to your mail or any data stored by FastMail. All access to your Dropbox is done using a random authentication token that can not be linked back to your FastMail account.

We’re pretty excited about this feature, and hope you like it!

Posted in News. Comments Off

Fast, full message searching across all folders

We’ve just rolled out a significant improvement to our search infrastructure.

Until now, searching for email in FastMail has been slow and idiosyncratic. Searching for words entered into the search box would only search the To/Cc/From/Subject headers of messages in the current folder. It wouldn’t search the message body content or across multiple folders. While there were options that made both of these possible, they were incredibly slow, making finding the messages you wanted frustrating.

Now though, when you enter words in the search box, we search the To/Cc/From/Subject headers as well as the entire message body content, and we do it across all messages in all folders. Additionally, when searching for a word, we’ll show a snippet of the message content with the search term(s) highlighted in the preview area. We’ll also search for multiple different forms of the word e.g. searching for “condition” will find condition, conditions, conditional, conditionally, etc.

Most importantly, by making sure all this data is indexed, the searches are very fast, even with a million messages across dozens of folders. In most cases, it should only take a few seconds to search all your email and we have plans in progress that should speed this up even more over the coming months.

We’ve also built an easy to access advanced search builder. When you click in the search box, an auto-complete popup will appear as you type. The final item in the pop-up is an “Advanced search” option. Clicking on this will show an overlay that lets you construct an advanced search without having to leave the screen.

If you find yourself repeating the same search frequently, you can save it by clicking the “Save” button next to the search in the sidebar. The search will appear with your list of folders. Keyboard power users can quickly select the search using the “g” shortcut folder finder, just like a normal folder.

Note: The new search currently only works with the new (AJAX) interface. If you’re using the “classic” interface or the mobile interface, you’ll get the old search system for now. We hope to fix this in the future. Also if you want the old search behaviour (including sub-string matching), use the substr:(sometext) operator as described on the help page.

Posted in News. Comments Off

New domains added

We’ve recently added some new domains to Fastmail. You can use these domains to signup new accounts, or as alias email addresses in existing accounts (if your account level supports aliases). The domains are:

  • fastmail.im
  • fastmail.mx
  • fastmail.se
  • fastmail.tw
Posted in News. Comments Off

Intermittent bayes db corruption resolved

This is a technical post that describes the history and recent efforts to track down a bug that was corrupting some users bayes databases. Fastmail users subscribed to receive email updates from the Fastmail blog can ignore this post if they are not interested.

Over the past few years, we’ve had sporadic reports of users bayes databases being corrupted and reset back to empty. When this happened, it would cause email delivery for that user to fall back to using the global bayes database, which decreased the overall accuracy of their spam detection until they retrained the database with more spam and non-spam messages.

I had tried multiple times to track down what was causing this issue, but each time with no luck. Each time the problem occurred, there was an error message in the logs of this form.

bayes: bayes db version 0 is not able to be used, aborting!

Often searching the internet for an error message will find other people that have had the same problem and tracked down the solution, but in this case it didn’t. Each time I tried to work through the code to see what was going wrong, I reached a dead end and couldn’t see any obvious problem.

Since the corruptions were very intermittent and losing a bayes database isn’t critical, doesn’t cause email to be lost or inaccessible, and can be rebuilt just by reporting email as spam/non-spam again, tracking this down was always a bit of a lower priority issue.

Recently though, after one more corruption report too many, I decided once and for all to track down what was causing it. Bit by bit over the course of several weeks, I added more and more logging information to the server code to track down where in the code the problem was occurring.

The logging results proved to be very odd. In the vast majority of cases it showed that writing to a particular database worked fine, but every now and then, it caused data to be lost. Eventually I managed to create a reproducible test case. It turned out to be very odd issue because performing a particular programming action with a database library worked fine the first 5 or 6 times, but on the 6th or 7th, it would cause data to become lost. Clearly something odd is happening in the lower level library code.

Fortunately there was a straight forward workaround to the problem, so I’ve now patched our code with the workaround, and over the last few weeks I’ve monitored the logs which show the original error message above has completely disappeared and no databases are being corrupted any more.

I’ve reported bugs to the underlying modules causing the problems, so hopefully long term they’ll fixed as well.

https://rt.cpan.org/Public/Bug/Display.html?id=83060

https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6901

Posted in Technical. Comments Off
Follow

Get every new post delivered to your Inbox.

Join 4,633 other followers