The average user of the Fastmail website is probably a bit different to most websites. Webmail tends to be a "productivity application" that people use for an extended period of time. So for the number of web requests we get, we probably have less individual users than other similar sized sites, but the users we do have tend to stay for a while and do lots of actions/page views.

Because of that we like to have a long HTTP keep-alive timeout on our connections. This makes interactive response nicer for users as moving to the next message after spending 30 seconds reading a message is quick because we don’t have to setup a new TCP connection or SSL session, we just send the request and get the response over the existing keep-alive one. Currently we set the keepalive timeout on our frontend nginx servers to 5 minutes.

I did some testing recently, and found that most clients didn’t actually keep the connection open for 5 minutes. Here’s the figures I measured based on Wireshark dumps.

• Opera 11.11 – 120 seconds
• Chrome 13 – at least 300 seconds (server closed after 300 second timeout)
• IE 9 – 60 seconds (changeable in the registry, appears to apply to IE 8/9 as well though the page only mentions IE 5/6/7)
• Firefox 4 – 115 seconds (changeable in about:config with network.http.keep-alive.timeout preference)

I wondered why most clients used <= 2 minutes, but Chrome was happy with much higher.

Interestingly one of the other things I noticed while doing this test with Wireshark is that after 45 seconds, Chrome would send a TCP keep-alive packet, and would keep doing that every 45 seconds until the 5 minute timeout. No other browser would do this.

After a bunch of searching, I think I found out what’s going on.

It seems there’s some users behind NAT gateways/stateful firewalls that have a 2 minute state timeout. So if you leave an HTTP connection idle for > 2 minutes, the NAT/firewall starts dropping any new packets on the connection and doesn’t even RST the connection, so TCP goes into a long retry mode before finally returning that the connection timed out to the application.

To the user, the visible result is that after doing something with a site, if they wait > 2 minutes, and then click on another link/button, the action will just take ages to eventually timeout. There’s a Chrome bug about this here:

http://code.google.com/p/chromium/issues/detail?id=27400

So the Chrome solution was to enable SO_KEEPALIVE on sockets. On Windows 7 at least, this seems to cause TCP keep-alive pings to be sent after 45 seconds and every subsequent 45 seconds, which avoids the NAT/firewall timeout. On Linux/Mac I presume this is different because they’re kernel tuneables that default to much higher. (Update: I didn’t realise you can set the idle and interval for keep-alive pings at the application level in Linux and Windows)

This allows Chrome to keep truly long lived HTTP keep-alive connections. Other browsers seem to have worked around this problem by just closing connections after <= 2 minutes instead.

I’ve mentioned this to the Opera browser network team, so they can look at doing this in the future as well, to allow longer lived keep-alive connection.

I think it’s going to be a particularly real problem with Server-Sent Event type connections that can be extremely long lived. We’re either going to have to send application level server -> client pings over the channel every 45 seconds to make sure the connection is kept alive, or enable a very low keep-alive time on the server and enable SO_KEEPALIVE on each event source connected socket.

robm@fastmail.fm

When you want to send a download file to a user based on a web request, it’s well known you can just set the Content-Disposition header to attachment to get the browser to download the content and save it locally on the users machine. Additionally you can add a filename parameter to control the filename displayed to users.

Content-Disposition: attachment; filename="foo.txt"

The problem comes when the filename contains non-english (really non-ASCII characters). RFC2231 defines a way of adding character set encodings to MIME parameters. Unfortunately support for this RFC is scattered, and browsers have implemented various internal hacks/workarounds (eg. % URL encoded octets). The situation is sufficiently complicated that someone came up with a comprehensive set of tests and there’s a good stackoverflow answer.

However looking over the test case examples, I realised that there appeared to be a solution that would work on all browsers except Safari quite well. The attwithfn2231utf8 test shows that all modern browsers except IE and Safari support the RFC2231 encoding. The attfnboth test shows that if you have a traditional filename parameter followed by a RFC2231 filename* parameter, IE and Safari pick the traditional parameter. The attwithfnrawpctenclong test shows that if you use % URL encoded octets in a traditional filename parameter, IE attempts to decode them as UTF-8 octets.

Putting that together, if you want to send a file called foo-ä.html, then setting a header of:

Content-Disposition: attachment; filename="foo-%c3%a4.html"; filename*=UTF-8''foo-%c3%a4.html

Will cause IE8+, Opera, Chrome, FF4+ (but not Safari) to correctly save a file named foo-ä.html. This should be easy to do with a URL escaping library that encodes UTF-8 octets not in the unreserved character set.

robm@fastmail.fm

I’ve now done that on our beta server, so if anyone has any problems, please let me know at robm@fastmail.fm so we can fix them before we release the change to production.

Starting Fastmail

Fastmail started around 1999 when Jeremy Howard prototyped the first version and brought me on board to help develop the website. While both of us were reasonably experienced programmers, neither of us had done any web development. When we looked at the way to develop websites – CGI scripts in those days – we didn’t really like the way they worked, so I ended up developing a web framework to help make the application development easier. We still use that framework today. While it’s a bit esoteric, it has served us well and includes some features that are fairly unique, even to this day.

Jeremy had come from a management consulting background, and so had spent a lot of time onsite with customers. While email was a very important part of getting work done, he found the options available very limited. In those days, people could either get basic POP email from their ISP, or free but slow and ad-laden webmail sites (eg. Hotmail and RocketMail) that had less features (no POP, SMTP and certainly no IMAP). So we decided to concentrate on being a premium email provider that targeted users that wanted a professional service with lots of features (multiple personalities and signatures, aliases, plus addressing, sub-domain addressing, filing and filtering emails, etc), a fast interface (no ads, no graphics), open access to their email (POP and IMAP support) and solid customer support – and were willing to pay for it.

Building and growing the service

The original implementation started on one server at rackspace.com, and we started to let friends use it. We then started to publicise it more widely, mostly via word of mouth and on the forum Edwin helped us set up at www.emailaddresses.com (now www.emaildiscussions.com). We offered the service as free while building out the application, but from day one we told users that it would be targeted as a paid service.

With only myself and Jeremy as developers, customer support, system administration and forum support, it took us a while to get to the stage where we were happy with the service and ready to bill users, which started in 2002.

One of the things that the founders tried to avoid was taking on external debt, and Jeremy used his savings to fund the initial development. Initially I was just employed as a programmer, but with the successful launch and growth of the service and the work that required, it was decided to form a company, and bring myself and another person (Bruce Davey) in as the owners. Later on, another person would also buy in to the business, creating an equal partnership of four people.

Over the next decade or so, Fastmail continued on a fairly simple path. We kept building up the service, adding features and trying to scale to more users. We tried a few times to bring on more programmers, though we had mixed success with that until we finally built a small and core group that has been stable for the last 5 years or so.

As a side note, a few years back a friend lent me a book called The Beermat Entrepreneur. If you’re thinking of starting a small business, especially IT related, I think this book is a great read. It may have first been written almost two decades ago (an eternity in Internet time), but even with my single data point, the advice in it really resonated, especially the parts about people.

It talks about the 5 core people, the entrepreneur and the 4 cornerstone people you need to cover technical skills, delivery (operational side of the business), sales and finance. I can really see how each of those roles is important to a business. Jeremy was definitely the driving entrepreneurial force, always pushing the big idea. Bruce was our financial controller who made sure that we sold things for more than it cost to run them (important if you want a business to survive!) and I was the technical lead.

I think initially we lacked the delivery cornerstone, and were lucky to find Bron who brought a structure and repeatability to our setup that allowed us to recover from our small disaster in 2006 (2-3 day outage for a big chunk of users) to build the incredibly reliable and solid infrastructure we now have to deliver our service. The one thing we should have done from the start was to have someone in charge of sales. After the initial success from word of mouth, I think we expected sales to keep accelerating. In hindsight, it’s clearly the thing that we never executed well.

Instead of trying to do sales our self, we tried to get other people to do it for us. We did that by creating a co-branding system that allowed people to resell the Fastmail email service under a completely different brand with a customisable look and feel. Unfortunately that was a failure. Setting up a new co-brander took too much work, and every co-brander we tried was worse at sales and marketing than we were. They often expected to be able to just set the brand up, and gain users with little or no effort.

Recent status and changes in the email market

Fastmail has always been a small company. Today there are still just 3-4 main developers (myself, Bron, Richard and just in the last year, Kurian), and a couple of support staff (Yassar, Vinodh & Larry) scattered around the world. Despite (or possibly because of) those limited resources, I think we’ve managed to build a great product with lots of niche and powerful features as well as loyal users. We’ve also been incredibly reliable, especially in recent years, as we worked hard to make sure that we learned all we could from the shortcomings that led to the outage event in 2006.

Fastmail has been nicely profitable for many years, but not spectacularly so. As the above paragraphs show, we’re mostly technical and support people, and we don’t have a marketing or sales department that can grow our customer base significantly. We did try recently, and I have to thank Jack Miller for the hard work he put in in that area, but unfortunately it didn’t work out. Realistically only business accounts can have a reasonable return on any sales investment, and we really should have developed our business product 3-4 years earlier than we did. We were slow to enter that market and behind the leaders.

Again I think this relates back to the lack of a sales cornerstone again. Rather than having someone testing to market to see what profitable areas we should be attacking, we probably preferred to spend our time building neat technical stuff, fixing that edge case bug, or doing that fun thing… we’re really just geeks that like working with technology.

In retrospect, I think we had to face facts a bit; we were a small fish with limited resources in a market that has become vastly more competitive in recent years. We also needed to invest a large chunk of time and money in updating our interface and adding new features, especially better mobile synching, if we were to take that next leap.

Approach by Opera

Coincidentally, it’s around the time that we were considering our options that Opera came along and started a conversation with us. Despite being half a world apart, there turned out to be a lot of fit between our companies. They use a lot of perl, as do we. They’re a company run by technology people, creating a product that’s loved by geeks, is highly customisable, has a loyal fan base, and despite its small size, punches above its weight. I think that summary pretty much describes Fastmail as well.

So the timing was right, and Opera have an interest in picking up email as a core competency as well as a number of ideas about what they want to improve and what they want to build. As the other Fastmail staff are also interested in new opportunities, we are all making the transfer and becoming Opera staff and are committed to working there for several years at the very least. There’s already plans for some staff to move to Norway to work, a change of life after 5 years of just the 3 of us in a single office (apparently the Norwegian lessons are paying off… Jeg vil gjerne et øl til).

Looking forward to the future

It will be an interesting change, and something new which I’m looking forward to. I’ve been working for Fastmail for 10 years now. It’s been a great time. I’ve loved building the product and the company. Like anything, there’s been ups (it’s fun developing a site that customers really love and tell you about) and downs (some people are addicted to being able to access their email at all times, and running a 24/7 email service means that if people can’t get to their email for even just 1 minute, you’ll start hearing about it). After 10 years, it’ll be strange having a boss again. I’ve met a bunch of the Opera people, and it’ll be really great working with them. I know the other staff are also looking forward to working with the Opera team.

It will also be great to be able to work with Neil Jenkins again, who is also going to work for Opera having nearly finished his university studies. He worked for us over a couple of summers, and basically designed the entire "new" web interface, all the HTML, CSS and JS. We’ve already got 80% of a whole new AJAX interface done (remember in programming though, the first 90% takes 90% of the time, the remaining 10% takes the other 90% of the time), and I’m looking forward to completely finishing that work.

Of course there’s lots of other work that we’ll be doing with Opera, some of which I can’t comment on at this stage, but some of which we’re already working on.

Over the past few years, Bron has probably become the worlds expert on cyrus, the IMAP server we use. Recently he’s been refactoring large sections of the code to make it even more stable and RFC compliant, and adding new code to reduce the amount of IO required (eg. make it faster), and radically reduce the amount of bandwidth required for replication, which will be very useful in a multi-data-center setup.

Richard has been working on making sure every part of Fastmail is Unicode compliant, which will allow us in the future to translate the interface to different languages, and will finally mean that non-english folder names can be displayed correctly in the web interface.

As mentioned in the original blog post about the sale, FastMail.FM will continue to run and grow as the reliable email service you’ve known for over 10 years. Opera have clear plans for the future, and can help provide significantly more resources to build a bigger and better infrastructure and provide even more features.

Rob Mueller <robm@fastmail.fm>

Please email me (robm@fastmail.fm) with feedback if you’re able to test this feature. Assuming there’s no issues, this will be rolled out to production soon.

However IBM also have a down side. Often their Linux support is limited to Redhat and SUSE installs and the kernels that go with them. In some cases that means that they only distribute binary blob drivers which only work with the particular kernel and version distributed by Redhat/SUSE. Because we like to use Debian Linux, and also to compile our own kernels from source with just the modules and features we need, a binary blob driver or a driver that only works with Redhat/SUSE kernels is unacceptable for us.

We discovered this problem the hard way with our first X3550 purchase, which came with a ServeRAID 8k-i controller. It turns out that controller was based on some LSI “fakeraid” chipset which needed a binary blob driver (can’t find the link right now) and thus we couldn’t get it to work sanely. We ended up returning the machines.

On the other hand, the ServeRAID 8k controller (note the difference, not the –i version) is actually completely different, and works fine with any Linux kernel with the vanilla open source AACRAID driver. We’ve bought almost a dozen machines with the 8k controller and they’ve all worked really well.

Now however, the X3550 and X3650 machines have been replaced by the newer X3550 M2 and X3650 M2 models, which update the CPUs and motherboards to use the newer Nehalem based CPUs, as well as replacing all the available ServeRAID controller options with new ones again

• ServeRAID-BR10i SAS/SATA Controller (3577)
• ServeRAID M5014 SAS/SATA Controller (3877)
• ServeRAID M5015 SAS/SATA Controller (Battery not included) (0093)
• ServeRAID-MR10i SAS/SATA Controller (3571)

We’re having trouble finding much information about any of these controllers, and what support there is in the vanilla Linux kernel for any of these controllers. If anyone has some good information about vanilla Linux for these controllers (or lack there of), please email me at robm@fastmail.fm

Update: Someone from the IBM Linux Technology  Center passed on the following useful information to us that others might also find useful.

• ServeRAID-BR10i SAS/SATA Controller (3577)

  LSI 1068[E] support has been upstream prior to 2.6.14 which means that this controller is support by all newer Ubuntu versions. The driver for this card is mptsas.

• ServeRAID M5014 SAS/SATA Controller (3877) / ServeRAID M5015 SAS/SATA Controller (Battery not included) (0093)

  Both of these are supported by the same megaraid_sas driver which went upstream in 2.6.27 (megaraid_sas version: v00.00.04.01-rc1 or newer), which means these controllers are supported by Ubuntu 9.04 (based off 2.6.28)

• ServeRAID-MR10i SAS/SATA Controller (3571)

  LSI 1078 support has been upstream sinse 2.6.16 which means this controller is also supported by all newer Ubuntu releases. The driver for this card is megaraid_sas (v00.00.02.04 or newer)

1. Email addresses surrounded by ‘ (single quotes) were handled poorly, partly disappearing, but still being left in the data sent back to the server
2. Email addresses where the “phrase” part had more than one , (comma) in it were split incorrectly. For example: “smith, john, help center” <johnsmith@example.com>
3. Email addresses where the “phrase” part had a angle-bracketed address were handling incorrectly. For example: “<johnsmith@example.com>” <johnsmith@example.com>

I’ve now completely rewritten the address parser, and put it on our beta server for testing. The new parser fixes all of the above problems, should be more resilient to odd address formats, and now allows you to do things like paste a list of email address with only spaces between them (rather than having to put commas) and it should tokenise them correctly.

If you come across any new problems, please email me directly at robm@fastmail.fm with details including an example of how to reproduce the problem and I’ll look into it.

Update 16-Apr-2009: Since there were no reported problems, I’ve now rolled out the new tokeniser to the production servers.

The beta site is available here – http://www.fastmail.fm/beta/ – you can login using your regular account username and password.

• Overall features
  • Professional look with cleaner layout, reduced clutter and simple icons to make common actions clearer
  • More consistent layout between screens (eg. navigation bar always available, common sidebar)
  • Javascript used to improve user experience, but still works fine without
  • More semantic HTML makes future stylesheet (look and feel) customisation easier
  • Significantly improved mobile display on modern mobile devices (eg. iPhone, Opera Mini, Opera Mobile)
• Mailbox screen
  • Much better keyboard support – navigate using keys j/k/x/o/Enter, action menu via . (fullstop), search email via / (slash), search folders via , (comma)
  • Better auto-sizing to screen size to display more of the message subject
  • Cross folder searching available
  • Advanced search syntax available (eg. from:john subject:dinner since:”1 week ago”)
• Message read
  • Much better keyboard support (like mailbox screen)
  • Attached messages shown inline
  • Attached images shown as thumbnails
  • Better integration with file storage to save attachments
• Compose
  • Address book auto-complete and address tokenisation
  • Improved HTML editor
  • Auto-saving of drafts
  • Background upload of attached files
  • Improved spell check
  • Separate reply/forward quoting options
  • Default font face/size for HTML email

For more details, please see our new interface wiki page. To report bugs/issues, please see our new interface bugs page. We encourage all users to give the new interface a go and report any issues on the bug wiki page.

We plan to eventually roll out the new interface to http://www.fastmail.fm so all standard logins use the new interface. Depending on feedback, that should occur in the next month or two. We plan to run the old interface for 3-6 months after the changeover, but will eventually decommission the old interface.

If you have any particular comments you want to make about the new interface, please email me directly at robm@fastmail.fm.

You can read more about Jabber here: http://www.jabber.org/

Our Jabber service is configured to allow S2S (server to server) communications, so you can use your email address as an instant messaging address and connect to users of other XMPP compatible chat systems (including google talk, livejournal, and of course jabber.org). Currently there’s no way to connect with proprietary messaging systems like MSN, Yahoo, AOL, etc. However there are a number of multi-protocol instant messenger clients (eg Pidgin, Miranda, etc) that will allow you to connect to multiple messenger services at once.

The main advantages of using the FastMail Jabber server over other services are:

• Your login name/email address is the same as your chat name, so there’s only one address people have to use/remember
• If you’re using a business or family account, it makes it easy for everyone in the family/business to chat to each other without needing to signup for any external service
• We can interoperate with any other jabber system (eg google talk, livejournal, jabber.org, etc)
• You can setup the service so that when a chat session is closed (or times out after 5 minutes), the chat log is emailed to you (login to the beta server at http://www.fastmail.fm/beta/ and go to Options -> Chat Settings). This is better than using your messenger clients history feature because the log is stored as an email in our IMAP server, so it’s replicated, backed up, and accessible anywhere

The main limitations of the FastMail jabber server are:

• Can’t interoperate with proprietary messaging systems
• Multi-user chat (chatrooms) aren’t available at the moment
• You have to use your main login email address as the jabber id. You can’t use an alias

For instructions on setting up various clients, see this wiki page: http://wiki.fastmail.fm/index.php?title=ChatService. Note that most instant messaging clients have poor error reporting. If they can’t connect, they often don’t report an error, or report a poor error message. If you can’t seem to connect, please check that you’ve got all the options set correctly, and that you’ve entered your password correctly.

If you experience any problems, please report them to robm@fastmail.fm. Make sure you include as much information as possible, such as your username, operating system, messaging client, error message, etc. Try and include a screen shot of the setup screen and/or any error message.

Despite the high specs, I found that after a week of using it that it started behaving oddly. Basically the keyboard would start being slow and whatever I was typing would start lagging. This was regardless of application (Mail, Notepad, Firefox, etc).

After that it got even worse, with intermittent “pauses” occuring anywhere from 1 second to 10 seconds. The pauses would occur when I was typing or even holding down a key. For instance, hitting the Ctrl key (to do a Ctrl-click on a link in Firefox to open in a new tab) would sometimes cause the machine to “freeze” for 1-10 seconds.

I pulled up a bunch of monitoring tools, but they didn’t show anything interesting. The machine was barely using any CPU. If I left the performance monitor up, normally it updates every second. But if I typed into another window, the performance monitor too would “freeze up” for 1-10 seconds.

So turning to the Internet, I found lots of reports of Vista problems, but nothing that seemed to be specifically like this problem… until I came across this:

http://www.randomsupport.com/rs/blogs/index.php?title=dell_laptop_slow_lagging_delayed_keyboar

The problem description sounded the same. Looking at the update at the end, I pulled up the Dell Mediadirect program, disabled the “Instant Office” feature, rebooted, and suddenly my keyboard is back to actually being useable!

This seems like a serious bug to me. And that post is from September 2007. I can’t believe that every Dell Vista laptop was and still is shipping with this bug?

Update: It seems disabling the “Instant Office” feature wasn’t enough. Over the next day the machine became slow and laggy to keyboard response again. Killing the PCMService.exe process and removing Dell Media Direct does seem to have permanently fixed the problem.

robm@fastmail.fm