Search currently broken on beta.fastmail.fm

We’re currently in the process of rolling out some new vastly improved search code (more details in a future post). This code will allow fast searching across all message headers (from/to/cc/bcc/subject) and message bodies, across all messages in all a users folders. Finally!

Unfortunately this process will take a week or two to complete, and in the meantime searching in the new AJAX UI on http://beta.fastmail.fm will be broken for most users. This doesn’t affect users using the regular interface at http://www.fastmail.fm

Sorry for any inconvenience, we hope to get it fully working ASAP.

Posted in News. Comments Off

Goodbye old.fastmail.fm

Summary

In early 2009 we rolled out an updated web interface to all users. This is the interface you currently see when you login at http://www.fastmail.fm as most users do.

To give users time to transition, we continued to let people login to the old pre-2009 interface if they wanted to by going to the web address http://old.fastmail.fm. We’ve continued to support this for the last 3 years, but as only a few users are still using this interface, we decided to shut it down. For the last 3 months there’s been a prominent message each time you logged into http://old.fastmail.fm that noted this, and we’ve now fully shut down http://old.fastmail.fm.

Important point: This only affects users that were explicitly going to http://old.fastmail.fm to login. Users that use the regular interface at http://www.fastmail.fm (the vast majority) are completely unaffected.

The description below is a detailed history of the old interface and includes technical details about how much things have changed since 2009 and why maintaining http://old.fastmail.fm is no longer feasible.

Goodbye old.fastmail.fm

It’s been a long road, but the old FastMail web interface has finally reached the end of its life.

You can always access your email at https://www.fastmail.fm/ or try our beta site at https://beta.fastmail.fm/.

If you want to stop reading here, the things you need to know are security concerns and it was about to break anyway. Two good reasons why now is the right time to shut the old interface down.

If you want to know some of the technical background and the technologies that we have moved through over the years, read on!

A new infrastructure

Looking back through our version control history, my very first commit was on 2004-09-20! The original web interface commits are from early 2000, though it was started before then.

We switched version control systems at some point during 2005 from CVS to Subversion, which made branching much easier – but imported all our history, so we can still look back at those early changes.

One of our major branches was a huge infrastructure switch from Redhat 7.3 to Debian 3.1 (sarge), which we worked on throughout the second half of 2005. This was all merged back into the main branch, and we converted everything over in early 2006.

http://blog.fastmail.fm/2006/01/02/one-web-server-now-running-new-infrastructure-code/

We upgraded to Debian 4.0 (etch) during May 2007, soon after it came out.

A new interface

In 2008, Neil Jenkins (who is so awesome Opera hired him even before they had decided if they were going to buy FastMail) worked as a contractor over the summer to design a more modern web interface which would take advantage of the new features in web browsers.

We branched the code, and it diverged quite considerably. Features like cross folder searching required major internal datastructure changes, and the new interface had hooks all through the code. Our plan was always to retire the old code eventually.

We released the new interface to beta at the end of 2008, and rolled it out to everyone in 2009.

http://blog.fastmail.fm/2008/11/27/help-beta-test-new-web-interface/

http://blog.fastmail.fm/2009/02/17/new-interface-being-rolled-out/

An incompatible upgrade

Then in 2009, Debian 5.0 (lenny) came out. Lenny shipped with apache2 and mod_perl2, and no longer supported apache 1.3 or mod_perl version 1. We put quite a lot of work into porting our codebase forwards to apache2. Since "old" was going away soon, we didn’t duplicate the work there.

So we installed the new web servers on lenny, and kept a couple of servers called "oldweb" still running etch. It’s amusing now to remember all the hoops I jumped through to allow automatic installation of either system.

About this time we also had machines with enough memory that 32 bit address spaces were wasteful, particularly on the IMAP servers. We moved to running 64 bit kernels with 32 bit userland.

New hardware

In 2010, the Opera sale happened. One of the early steps was to replace some of our aging hardware with equipment that was better understood and supported by the Opera sysadmin department. This meant a new bladecentre for the non-storage systems (including web)

For a little while I had two blades (redundancy!) running "oldweb" code. That’s a huge amount of very under-utilised resource.

And, to be honest, managing new blades with ancient OS was a pain. Things didn’t work well. The configuration tools we built for the new hardware didn’t run on etch.

When we moved to Debian 6.0 (squeeze) and at the same time went fully 64 bit, it was time to do something about "old".

We also moved version control systems AGAIN in late 2010 – from subversion to git. The old web servers were left on subversion, because they weren’t getting much in the way of changes any more. One more "split" in how things were done.

Fully virtual

Rather than having to support "real hardware", I built an etch virtual machine. Everything else was running squeeze 64 bit, but we still had a full 32 bit etch install path just to support oldweb.

While all this was happening, there were occasional changes required to support changing database schemas, configuration mechanisms, and interaction with other parts of our system. At some point I just took a snapshot of the current tree and started a new git repository so we could archive the subversion server entirely.

Maintaining the virtual machines was a real pain though. They were run in the background on some of the web servers to free up the hardware for more demanding tasks. This meant changing the network interfaces to be bonding drivers, custom configuration, lots of pain. There were occasionally long outages as we changed things and then had to patch oldweb to catch up.

Worst of all, we were maintaining the ENTIRE stack – support daemons, log rotation, pop fetching…

Old lost features over time – we just couldn’t keep them working, so we ripped the code out. Particularly some of the more advanced configuration screens – and everything related to billing.

Single component

In the end the virtual machines were too much work. Our authentication system in particular had many changes under the hood, and it just wasn’t going to keep working. We had a couple of really bad problems with file storage, where we were sure that something "couldn’t happen", but then it turned out old was still doing things differently. Talking to the wrong databases, running the wrong queries. We seriously considered dropping old at that point, but I wanted to give it a bit longer.

So I build a chroot installation of etch on our web servers, and bind mounted the daemon sockets into the chroot. This allowed us to run just the web interface code itself on the old branch, while running everything else in the modern, managed, outside world. I built a custom init script which could set up all the necessary mountpoints (/proc, /dev, /var/run, even the tmpfs with mmaped caches was shared) – and forward ported more of the code.

This was built with debootstrap originally, but in the end it was getting unreliable even fetching etch packages, so I build a .tar.gz file with the filesystem for the chroot, and a fresh install just unpacked that. As we changed internal config systems, I kept "oldweb" up to date. A couple of commits every month.

So that’s brings us to today. An init script (apache-oldweb), a chroot environment with a snapshot of a Debian etch machine with apache 1.3 and mod_perl version 1 – running perl 5.8. Everything else is perl 5.10 or newer, so I even have to backport some idioms as I bring back the bits which it just can’t live without.

I have done basically all the "keeping old alive" for the past couple of years – for a smaller and smaller set of users who still log in there. Backporting everyone else’s changes as they impacted old.

And etch doesn’t have security support. Hasn’t for ages. Sure it’s in a chroot, but it still has access to everything.

The final straw

But there’s one thing which oldweb can’t survive. We are redesigning how our session management works. There are some great benefits – bookmarkable URLs, remote logout of stale sessions, reduction of password typing on annoying little smartphone keyboards.

Everything will change, and old would have just stopped working. It’s not worth the changes to make it work. Particularly with the larger gap between the two systems as time goes on.

Also, and even worse, old interface is exposed to the wider internet – and it has full read/write access to the database and all emails. If there are security problems, all our users are at risk – not only those who use it directly.

It’s no longer safe, and it was going to break beyond easy repair in a few days anyway. It is time.

Goodbye oldweb

Posted in News, Technical. Comments Off

Now enforcing SSL/TLS encryption on all IMAP/POP/SMTP/LDAP connections

As we’ve noted over the last two months, we’ve been planning to enforce SSL/TLS security on all IMAP/POP/SMTP connections. All users that needed to change their settings have been notified multiple times. Additionally, over the last few weeks, we’ve also notified all LDAP users that need to change their settings to enable SSL/TLS as well.

We’ve now done this and have completely disabled all non-SSL IMAP/POP/SMTP/LDAP ports. This means you can no longer access any services on the ports 110, 143, 25 or 389.

Over the next few months we’ll also be looking at enforcing SSL/TLS encryption our remaining services such as XMPP and DAV. We’ll post more as we work out our approach to enforcing SSL/TLS for these services.

Posted in News. Comments Off

Enforcing SSL/TLS encryption on all IMAP/POP/SMTP connections from July 14

As noted previously, we’ve been planning on enforcing SSL/TLS encryption of all communication between a user’s email software and our servers, ensuring that no one can eaves drop on your username or password to steal your login credentials.

Over the past month and a half we’ve been sending notification emails to all users using insecure connections with details of how to fix their email software configuration. During that time the vast majority of users have already reconfigured their email software to correctly use secure and encrypted connections.

At some point we have to completely disable the insecure connection ports and we’ve decided to do that on July 14. From July 14 onwards, only the official configurations described on our Server Names and Ports help page will be active and supported, all other ports and hostnames will be disabled.

The vast majority of users are already using the correct configuration and won’t be affected. Only the small number of users still using insecure configurations (despite the regular notifications we’ve been sending them for the last month and a half) will have problems. We’re now contacting those users to let them know they have only 5 days left to change their configuration.

Posted in News. Comments Off

Changes to web interface Address Book rolled out

We’ve just rolled out a few changes to the address book available in the web interface. These changes are based on some analysis we did of how people are using the address book.

  • Remove the "Description" field on address locations

    We found that very few people used this field (the vast majority were blank), and for those that did put something in here, it was usually duplicate information (either just the string "Home", "Work" or "Other", and just duplicating the selected address type) or confused information (duplicating the first line of the address itself). So we’ve removed this, and in the few cases it appeared to be used, we’ve moved the information into the first line of the address itself.

  • Remove all custom fields

    Few people were using custom fields, and in the majority of cases people were actually putting data in here that should have been in another location. Most custom fields were some form of phone number and those should clearly be in the phone contacts section. The likely reason for this happening is because the previous interface didn’t make it obvious where to put phone numbers, which we’ve now also made clearer (see below). The other use of custom fields was for new services like Skype and Twitter. We’ve added new contact types for those services.

    Any existing custom fields have been moved to the appropriate phone/email/online contact type, or where we couldn’t identify an appropriate type, we’ve moved the data into the Notes section.

  • Add new contact types for Skype and Twitter

    Apart from the phone types, these were clearly the most used custom field types, so we’ve added these as explicit online contact types.

  • Split the old Contacts section into 3 separate sections: Email, Phone and Online contacts

    Because we’ve always allowed an arbitrary number of "contacts", there was a single Contacts section where you could select the contact type you wanted to add: Email, Phone, Web, Instant Messenger, etc. However because the selection  of which type to add was via a pop up menu which defaulted to "Email", it wasn’t actually obvious that you used the same section to add phone numbers, web addresses, etc.

    So we’ve now split this into three separate sections for "Email", "Phone" and "Online" contact types.

Based on our analysis, we believe these changes make the address book easier to use and also better matches the actual data people are wanting to see and store, while removing unneeded and rarely used complex or difficult to understand features.

Posted in News. Comments Off

Changes to webmail login

TL;DR: We’re now making all connections to the Fastmail web interface immediately redirect to a secure (https) connection.

As part of our commitment to making all connections between users computers and our servers secure and encrypted, we’ve just made some changes to our webmail login page. In most cases, users won’t notice any change because we made Secure Login the default almost a year ago. The new changes will only affect the small number of users that have special login requirements.

The main change we’re making is that where previously we would redirect from an insecure (http) to secure (https) connection during login, or on returning to Fastmail on a computer you’d logged in via before, we will now redirect to the secure login screen immediately when you connect to Fastmail. That is, as soon as you go to http://www.fastmail.fm (insecure) or http://www.sent.com (insecure), we’ll always redirect to https://www.fastmail.fm (secure).

Going to other https:// domains that aren’t supported (e.g. https://www.sent.com, a secure connection, but will report a certificate error) will redirect to https://www.fastmail.fm as well.

This will also be the case for businesses and families that use their own domain for logging in (e.g. http://mail.digitalintegrity.com), they’ll also be redirected to https://www.fastmail.fm, but we will continue to correctly show the family/business login screen.

There are a couple of additional exceptions to this.

The mobile UI domains that start with the http://m. prefix like http://m.fastmail.fm (insecure) and http://m.sent.com (insecure) will redirect to https://m.fastmail.fm (secure). This will always show the mobile login screen and mobile interface when you login.

The special "sticky ssl" domains that start with the https://ssl. prefix like https://ssl.fastmail.fm (secure) and https://ssl.sent.com (secure, but certificate warning) will "stick" to that domain. This may be useful as a work around for some proxies that block hostnames with the word "mail" in them.

If for some reason you need to use an insecure login (which we highly recommend you do not do), you will explicitly need to go to the URL http://insecure.fastmail.fm. If you use this to login, data sent between your computer and our server will travel unencrypted over the Internet. This service is only provided for dire circumstances, is highly discouraged, and may be removed in the future.

For the curious, here’s a list of all the transitions that should happen. The "(W)" means you’ll see a certificate warning about mismatched hostnames.

https://www.fastmail.fm               -> stays at https://www.fastmail.fm
http://fastmail.fm                    -> https://www.fastmail.fm
http://sent.com                       -> https://www.fastmail.fm/?domain=sent.com
http://www.fastmail.fm                -> https://www.fastmail.fm
http://www.sent.com                   -> https://www.fastmail.fm/?domain=sent.com
https://fastmail.fm                   -> https://www.fastmail.fm
https://sent.com (W)                  -> https://www.fastmail.fm/?domain=sent.com

http://mail.digitalintegrity.com      -> https://www.fastmail.fm/?domain=digitalintegrity.com
https://mail.digitalintegrity.com (W) -> https://www.fastmail.fm/?domain=digitalintegrity.com

http://m.fastmail.fm                  -> https://m.fastmail.fm
http://m.sent.com                     -> https://m.fastmail.fm/?domain=sent.com
https://m.fastmail.fm                 -> stays at https://m.fastmail.fm
https://m.sent.com (W)                -> https://m.fastmail.fm/?domain=sent.com

http://ssl.fastmail.fm                -> https://ssl.fastmail.fm
http://ssl.sent.com                   -> https://ssl.sent.com/ (W)
https://ssl.fastmail.fm               -> stays at https://ssl.fastmail.fm
https://ssl.sent.com (W)              -> stays at https://ssl.sent.com/ (W)

http://insecure.fastmail.fm           -> stays at http://insecure.fastmail.fm
http://insecure.sent.com              -> stays at http://insecure.sent.com
Posted in News, Uncategorized. Comments Off

New domain: fastmail.nl

We’ve just added fastmail.nl (.nl is the TLD for the Netherlands) to the list of our available domains. That means you can can now signup an account or create an alias on the Options –> Aliases screen (subject to your account service level) in this domain.

Along with our primary domain fastmail.fm, this adds to our existing list of available “fastmail” TLDs.

fastmail.cn
fastmail.co.uk
fastmail.com.au
fastmail.es
fastmail.in
fastmail.jp
fastmail.net
fastmail.to
fastmail.us

Posted in News. Comments Off

Enforcing SSL/TLS encryption of all connections

Users regularly tell us how important the security and privacy of their email account is. Sometimes because of how their email software was initially configured, users don’t realise that their username and password are being sent over the Internet unencrypted, which is often a genuine surprise and concern.

Because of this, we have decided to enforce that all communication between a user’s email software and our servers is encrypted, ensuring that no one can eaves drop on your username or password to steal your login credentials.

If we detect that you are currently using an insecure (non-SSL/TLS) connection to send or receive email, we will send you a notification directing you to this page which explains how to fix your email software. You will keep receiving this message until you have successfully fixed your configuration.

We will be rolling out these changes over the next few weeks and will give people until the end of June to change their software. We believe these changes are in the best interests of all users and are modern best practice on the Internet these days.

Posted in News. Comments Off

Understanding SSL vs TLS vs STARTTLS

There’s often quite a bit of confusion about the different terms SSL vs TLS vs STARTTLS. To help explain the differences and a bit of the history behind these terms (especially with regard to email protocols), I’ve put together a help page that I hope is useful for people.

http://www.fastmail.fm/help/technology_ssl_vs_tls_starttls.html

Posted in News. Comments Off

Singapore proxy server discontinued

Some years ago, when connectivity within the pacific region was less reliable, we added a small proxy server in Singapore which forwarded sessions down a VPN connection to our datacentre in New York.

The world has moved on, and this service is barely used. Reading the logs it’s almost all search engines scanning our help pages, which is just going to direct people to the slow proxy copies rather than the originals.

So as of today, the sg.* hostnames point directly to our main New York addresses, and the Singapore proxy will be shut down.

Posted in News. Comments Off
Follow

Get every new post delivered to your Inbox.

Join 4,623 other followers