New IP addresses in Iceland

We have just transitioned our Iceland servers to a new network range (belonging to us rather than to Opera).

This means that the addresses of ns2.messagingengine.com and in2-smtp.messagingengine.com have changed.

Unless you have hard-coded the old addresses somewhere, you shouldn’t see any difference. The new addresses will propogate over the next hour or so.

Posted in Technical. Comments Off

FastMail changelog update

The following changes have been rolled out to production:

  • The keyboard shortcuts for reply/reply all have been reverted to “R” and “A” respectively to match the shortcuts used by other services.
  • Preference added to change the default reply behaviour from “Reply to all” to “Reply to sender” (the preference is on the Settings -> Preferences screen, at the bottom of the “Writing” section).
Posted in Feature announcement. Comments Off

FastMail changelog update

The following changes have been rolled out to production:

  • New manual fetch button for POP links on the Settings -> Accounts screen.
  • When you manually refresh a folder (click it in the sidebar, or press ‘u’ ), any POP links that fetch into that folder are also refreshed.
  • Reply to all is now the default button shown at the top of every message. You can reply to just the original sender via the "More" menu, or the buttons at the bottom of the message.
  • Keyboard shortcuts for replying have changed. In line with our new default of reply-to-all, "R" is now *Reply to All*. "Shift-R" is now "Reply to Sender". "A" has been removed as a shortcut.
Posted in Feature announcement. Comments Off

Making FastMail even more secure

We have a great responsibility at FastMail to keep your email secure. As part of our continual process to maintain and improve security, we recently made a number of changes. This blog post summarises what we’ve done and why it makes your email even more secure. It’s also a useful checklist of security measures to think about if you run your own web application.

This is a technical post for information only. You do not need to make any changes to your FastMail account or your email software.

Preventing cross-site logins

Cross-site request forgery is a common security problem on the internet. Briefly, if you are browsing a malicious website A, that page can make a GET or POST request to any other website B, which includes any authentication cookies you have for website B. The page making the request from website A can’t see the response, but if the request changes state on the server it could compromise the account, for example by forwarding all of your mail to the attacker’s account.

FastMail has long had robust CSRF protection throughout our application. However, one thing we didn’t protect from CSRF was the ability to log in to an account. At first glance, an attacker being able to log someone into the attacker’s own account doesn’t seem like much of a security problem. However, there are two issues that can arise from this:

  1. An attacker could manage to confuse the target into sending an email from the wrong account. This is easy to do if you’re not expecting yourself to be logged in elsewhere. The email could then be read by the attacker.

  2. Any stored XSS vulnerabilities that only affect the user’s account suddenly become much more exploitable. For example, suppose a user’s folder name was not properly escaped somewhere: it could be used to inject a script into the page. Since only the user can modify the folder name, and the pages that show that folder are only accessible when logged in as that user, this does not help an attacker target anyone but himself. However, if the attacker can log someone else into his account and force the script to run, then he could potentially access any other accounts that the target is currently logged in to.

Given these risks, we’ve now taken steps to prevent logging in to a FastMail account from anywhere other than our own login form. Since on the homepage we don’t yet have a session (at least not one that lives longer than a couple of minutes), we’re doing strict referrer checking instead of using a CSRF token. The “Referer” header in the POST request must be present and must match the protocol and host (normally https://www.fastmail.fm), otherwise the login is rejected. (And yes, the referrer header really was standardised with an incorrect spelling.)

Whilst there are several ways of stripping a referrer header, as far as we are aware there is no way to fake one without using a browser extension or the like (at which point the extension could do anything it likes to the page, and no amount of CSRF tokens is going to save you).

Moving all user websites on to user.fm

FastMail has long had a feature where you could host some files or a small website via your FastMail account. For the account joebloggs@fastmail.fm, for example, we offered the ability to host files at joebloggs.fastmail.fm.

Unfortunately, this provides another way to log a user into your own account, because cookies are not subject to the same security restrictions as other web features. A user webpage cannot read any of our session cookies (on the http://www.fastmail.fm subdomain), so it cannot steal a session, however it can write its own cookies to the fastmail.fm domain. There’s no way for our server to tell that these were not created by us on the http://www.fastmail.fm domain. If you log into your account on your computer, you could then take the cookie and write it to someone else’s computer via your website, effectively logging them into your own account.

To resolve this, we’ve moved all user content on to a new domain, user.fm, and set up redirects so existing URLs will continue to work. For example, joebloggs.fastmail.fm will now redirect to joebloggs.fastmail.fm.user.fm.

Please note: this does not apply if you have your own domain. That will continue to work just as before. The move is only for content at one of our domains.

Content security policy

Cross-site scripting (more commonly known as XSS) is by far the most common type of security vulnerability to occur in web applications. An XSS vulnerability allows an attacker to inject a script into a web page on the target site, allowing them to access any data the user is currently authorised to access.

Arguably one of the biggest steps towards improving the security of web applications in recent years is the introduction of the Content-Security-Policy (CSP) header. You can use this to control what resources a modern browser will allow the webpage to load and run. We’ve been testing this on our beta server for some time, and we’ve now rolled it out to production. The CSP we are using for most of our webmail pages is the following:

default-src 'self'; script-src 'self' 'unsafe-eval' https://api.pin.net.au;
style-src 'self' 'unsafe-inline'; font-src data:; img-src *;
media-src 'none'; object-src 'none'; report-uri /log/csp

The most import thing about this policy is that it does not allow inline scripts to run, or for scripts to load from domains other than our own (with a specific exception for our payment provider). This means that should a future XSS bug be discovered, an attacker cannot exploit it against you if you are using a modern browser: the only scripts that can run are those hosted on our own domain (which are all written by us and known to be safe).

The other big change we’re going to be making next week is to move most user content off the fastmail.fm domain. When you open an attachment it will load in the fastmailusercontent.com domain instead. This is an isolation measure to provide an extra layer of security via the browser cross-origin restrictions. At the moment, we use a Content-Disposition: attachment header to force the browser to download rather than open any potentially dangerous files (all files except for a few specific safe MIME types such as JPEG images). Because each attachment will be isolated on a different domain (with a per-message authorisation key), we will also be able to allow more types to be opened in the browser, as each attachment is effectively sandboxed and cannot access sensitive data.

Bug bounty

Like many other internet companies, we recently introduced a security bug bounty program as a way of encouraging responsible disclosure and rewarding security researchers for their hard work.

We’d like to say a public thanks to the first two recipients of our bounty: Prasant Sharma for disclosing a stored XSS vulnerability in our support ticket system, for which we awarded $1000, and V. Harish Kumar for finding two minor self-XSS vulnerabilities, for which we awarded $200.

We welcome all whitehat security researchers who wish to help us maintain our high security. Full details of the rules and how to submit a vulnerability are available on the security issue reporting guidelines help page.

Other measures already in place

The following additional security measures have been in place for some time, but we’ve not blogged about them before.

Secure, HTTP-only cookies

All session cookies are marked “Secure” and “HTTPOnly”. This means the browser will not send them over an unencrypted connection, and they can not be read using JavaScript. This means that should an XSS attack occur, the attacker can only steal data whilst the page is still open on the victim’s computer: he cannot steal the cookie to transfer the session to another computer.

HTTP Strict Transport Security

We send the following header back with all of our responses:

Strict-Transport-Security: max-age=31536000

This tells all modern browsers to, from that point on, only connect to us over HTTPS, even if the user types in a non-HTTPS URL. This reduces the chance of a man-in-the-middle attack intercepting an insecure connection and proxying the user, instead of redirecting them to the secure site.

Frame protection

With basic support even in browsers as old as Internet Explorer 8, the X-Frame-Options header is the easiest and best way to prevent click-jacking and other attacks based on embedding your web app inside a malicious site. By setting the header X-Frame-Options: SAMEORIGIN, we only allow our pages to be framed by other pages on our domain.

Disabling content type sniffing

Content type sniffing is where browsers ignore the MIME type sent by the server and interpret the content as something else. This can be dangerous, as something that the server thought was safe (such as an image) could be interpreted by the browser as an HTML page, opening you up to XSS attacks. We disable this behaviour by setting the X-Content-Type-Options: nosniff header.

Working towards a secure web

We strongly believe in making the internet more secure, and one of the ways we feel we can help make this happen is to publicly document more of the security measures we have in place. This serves to both help educate and inform other sites, and to allow public scrutiny to improve our processes.

Thanks for reading, and as ever, thanks for using FastMail.

Posted in News. Comments Off

SSL certificates updated again

A few days ago we updated our SSL certificates. The algorithm used to sign these certificates (SHA256) presented problems with some older clients and operating systems, notably WebOS and Nokia devices. To fix this we got our CA (DigiCert) to re-sign the certificates using the older SHA1 algorithm, which should work pretty much everywhere. These certificates are now live on all of our domains.

Most users should not notice any change. If you are on a device or client where you’ve had to install the DigiCert root certificate in the last few days, you may need to do this again as these certificates are signed from a different root certificate. If that affects you, the root certificate is available from DigiCert’s root certificate page and is called “DigiCert High Assurance EV Root CA”. If you also need the intermediate cert, its available from the same page with the name “DigiCert High Assurance CA-3″.

Posted in Technical. Comments Off

Please update your FastMail password

We’ve just sent the following announcement email to all FastMail users.


Dear FastMail User

You may have heard of a recent security bug in the OpenSSL library (that has been called ‘Heartbleed’) used by two-thirds of the Internet including ourselves and other major sites like Amazon, Google, Yahoo, etc. FastMail was quick to update its servers to fix this bug and issue new SSL certificates as soon as we were made aware of it.

We have no reason to believe any of our servers were targeted or exploited by this security flaw, but given the nature of the flaw it’s impossible to know if this bug was being exploited before it was announced.

Because of this, we are recommending that all FastMail users logout of all existing sessions and change their account passwords.

Again, there’s no evidence our servers or your password have been compromised, but we’re recommending this as a precautionary measure.

If you hate remembering passwords, we recommend you use a password manager program to remember them for you. Most modern browsers (e.g. Firefox, Chrome, etc) have a password manager built in and will offer to remember your passwords for you. LastPass and 1Password are also popular choices.

When you choose a new password, it’s important that you do not use the same password elsewhere and choose a password with reasonable complexity.

Your email is often the key to your online world. Many sites let you reset your password by sending a reset code to your email address. When you reuse your FastMail password at other sites, you’re making it much easier for attackers to potentially break in to your email account. Other sites often don’t have the same high security measures as FastMail (such as compulsory HTTPS, locked-down servers, etc.), which makes them much easier for criminals to break in to. If they hold your email address and the same password that you use for FastMail, the attacker can then access your email account and get into everything else you use online.

If you’re using alternative logins already, we recommend you delete and re-add them with any base password changed.

To change your password and log out of all existing sessions, you can use these steps.

Change password in current interface

  1. Log in to your FastMail account using the web interface
  2. From the menu at the top left, select ‘Password & Security’
  3. Enter your existing password where directed
  4. Enter your new password where directed. Re-enter again to make sure we got it right
  5. In the ‘Logged in Sessions’ section, click ‘Log out’ next to each existing session
  6. Click ‘Done’ to dismiss the panel
  7. From the menu at the top left, select ‘Log out’
  8. Now log in to your account again with your new password. This is often useful as a password manager will now prompt you to remember your password at this point.

Change password in ‘classic’ interface

  1. Log in to your FastMail account using the web interface
  2. Select the ‘Account’ item at the top right
  3. Select the ‘Password/Security Settings’ item
  4. Enter your new password where directed. Re-enter again to make sure we got it right
  5. Enter your existing password where directed
  6. Click ‘Update Password’
  7. Click ‘Logged In Sessions’ in the sidebar on the left
  8. Click ‘Delete’ next to each existing session
  9. Click ‘Log out’ at the top right
  10. Now log in to your account again with your new password. This is often useful as a password manager will now prompt you to remember your password at this point.

Again, this is a highly precautionary measure. FastMail is extremely concerned about security and has always tried to be highly pro-active with keeping our customer’s accounts and data as secure as possible.

Regards,

The FastMail Team

Posted in News. Comments Off

When two-factor authentication is not enough

TL;DR: This is the story of a failed attempt to steal FastMail’s domains.

We don’t publish all attempts on our security, but this one stands out for how much effort was put into the attack, and how far it went.

We’ve had a handful of minor attack attempts recently. Targetted phishing emails to staff trying to steal credentials. An NTP-based DDOS which was quickly mitigated by NYI, our excellent hosting service.

These sorts of attacks are the “background radiation” of the internet. Along with port scans and entries in the web server logs from malware trying us out to see if we’re vulnerable to old PHP bugs (hint, we’re not). It’s the reality of being on the internet.

This blog post was first drafted before the Heartbleed fiasco. Sometimes, no matter how careful you are, you get a nasty surprise. We responded very quickly, as always. Anyway, on with this story.

About a month ago, our hostmaster@fastmail.fm account suddenly wound up subscribed to hundreds of mailing lists. All these mailing lists failed to use double or confirmed opt-in, so someone was simply able to enter the email address into a form and sign us up, no confirmation required. This really is poor practice, but it’s still pretty common out there. A special shout-out goes to government and emergency response agencies in the USA for their non-confirmation signup on mailing lists. Thanks guys.

The upshot was that the hostmaster address was receiving significant noise. Rob Mueller (one of our directors) wasted (so we thought) a bunch of his time removing us from those lists one by one, being very careful to check that none of the ‘opt-out’ links were actually phishing attempts. This turns out to have been time very well spent.

Internet identities

At FastMail, our security is central to the safety of our users. Given access to an email account, an attacker could reset passwords on other sites, including those which allow spending real money.

We take this responsibility very seriously, and we’re always looking for ways to improve our security.

Two factor authentication (2FA)

The Domain Name System is one thing that’s even lower layer and more central to identity security than the email server itself.

Based on recent articles in the tech press, we really wanted to have ALL our domains protected by two factor authentication.

Our domains were historically spread amongst multiple registrars. We chose to consolidate with Gandi because they have a great slogan (“no bullshit”), they support 2FA, and they support all the top level domains we require.

Robert Norris, one of our sysadmins, was in charge of the migration. He set up a corporate account with Gandi to get assistance in transferring the domains, and set up two-factor authentication at the same time.

Gandi uses the popular OATH TOTP (also known as “Google Authenticator”) mechanism. Rob wrote a small TOTP client and placed it with the key on our management servers in the secure storage area where we also keep our SSL certificates. The account password itself was encrypted in our password manager, which is stored separately.

Only a small number of trusted people have access to the credentials for our Gandi account. We are satisfied that this level of security is strong enough.

The attempt

On March 19th, this came to hostmaster@fastmail.fm:

From: Gandi Corporate Team
To: hostmaster@fastmail.fm
Subject: [RN1374-GANDI] Email address update request
Date: Wednesday, March 19, 2014 8:27 PM

Hello,

We received an email update request for the account RN1374-GANDI.

Previous email address: hostmaster@fastmail.fm
New email address: fastmail.fm@qq.com

If you are opposed to this modification, thank you for letting us know
only by replying to this email.

If you can read this message, then you can recover the password of your
account, and thus modify the email address of the handle. In that case,
we won't take care of your request.

Without any reply from your side, we will proceed within 24 hours.

Best regards,

Gandi Corporate

The hostmaster alias actually forwards to three of us, and we were all hyper-alert, so we thankfully noticed this email.

Within twenty four hours.

One day.

Gandi assure us that their fraud detection systems would have detected this, but for the 2 weeks it took from this email until we had full control over our account again, we were worried.

This request had completely bypassed our two-factor protection.

Forged source addresses

There is a well known problem in network security. You can’t trust the source address of an IP packet – they are trivially forged.

It’s the reason why we have source port randomisation, sequence number randomisation… all the things designed to stop an attacker being able to forge both an initial SYN packet and also the response to an ACK packet to bring up a TCP connection.

While they can falsify the source of a request, an attacker without full network control can not receive the response to their forged packet and continue the handshake.

This is why this email was such a surprise. Like the poor quality mailing lists mentioned above, it didn’t require a confirmed opt-in. We had to reply to say that we didn’t want the contact email address changed.

This means that a forged source address was sufficient. Even though the attacker couldn’t read email to hostmaster@fastmail.fm, they didn’t need to. All they needed was for us to not read it.

To Gandi’s credit, they responded very quickly to our “NO, DON’T CHANGE IT” email, and locked our account to stop any further shenanigans while they investigated and collected more documents from us.

Falsified documents

We discovered that Gandi received a paper email change form (pdf) claiming to be from a “Robert NORRIS” (the name which appears on our whois data), along with pictures of a passport of said “Robert NORRIS” and company registration documents also claiming to be for FastMail Pty Ltd.

At the time of writing, we are still in debate with the Gandi Legal Department about whether they can even show us these documents. They claim that French Law forbids them from showing us documents which purport to be from us. This is something to be aware of when choosing an vendor – different companies operate in different jurisdictions. There’s also a certain degree to which the conservatism of legal departments (protect the company as much as possible) conflicts with the corporate motto (“no bullshit”). The first response we got was certainly bullshit – “in order to meet a legal or regulatory obligation”. We challenged them to give an actual legal obligation and were given Article 226-15 of the French Criminal Code, along with rough English translation as follows:

“The act, committed in bad faith, of opening, deleting, delaying or diverting correspondence, whether or not it arrived at its destination, and addressed to a third party, or to fraudulently gain knowledge thereof, is punishable by one year of imprisonment and a fine of 45,000 EUR.”

We don’t believe that law is relevant – it’s the “no interception” law that exists everywhere, and doesn’t forbid anyone from quoting documents in replies to the purported source of those documents. If the law really was as Gandi Legal seem to be interpreting it, it would be illegal to quote an email in your response unless you were certain that the source address hadn’t been faked.

Was this a “security flaw”?

Security is built in layers, and I would definitely say that the fact that we received that email means one of the layers was weaker than it should be. Partly it’s poor choice of wording (Gandi claim that they would not necessarily have changed the email within 24 hours, depending on other investigations).

It still would have been necessary to either disable or reset the two-factor authentication on our account as well for the attacker to get full control. That’s difficult, but not necessarily impossible. After the fact, there’s no way to know how it would have gone down. We certainly weren’t willing to take the risk of doing nothing and seeing what happened!

What we do know is that the attacker was very determined, and willing to go as far as forging documents while simultaneously generating noise to make us less likely to notice the attack. They must have figured they had a chance.

Improving security for the future

A disadvantage of adding something like two-factor authentication after the fact is that you may miss the interactions with your existing processes. Gandi’s paper “email reset” form makes a lot of sense in the world where most of their customers are individuals or small businesses with one or two domains, and using addresses that they may lose access to. With no other factors, if they lose access to the email address and forget their password, there needs to be a process to regain access.

It’s always great to have a consistent process. Having a consistent process means that attackers can’t just try their luck until they find someone who is more trusting than average. Australia has a fantastic system called the 100 point check for authenticating people. We like process, consistently applied.

The problem we have is that we didn’t expect that the account email address could be changed without any reference to our two factors at all. Maybe nobody at Gandi realised either. That’s a security flaw – even if it doesn’t mean everything is totally broken.

We have had some very frank discussions with Gandi over the past week, and they agreed to make all three of the improvements we proposed as a result of these events:

  • the setting “disable password resets via email” was not on the security settings page of their website. Because of this, we hadn’t discovered and enabled it. They are moving it to the security page.
  • if an account has 2FA enabled, a red flag will automatically be raised against the request, meaning significant extra investigation will be done.
  • if an account has 2FA enabled, then an active confirmation will be required from the owner of the account before changing the email address. This means it will be harder to regain access if you lose all your factors, but that’s a good thing! Turning on 2FA means you want it to be hard for anyone who doesn’t have those two factors to gain access.

These steps will make attacks against Gandi accounts even more difficult in future, and we applaud their efforts to improve security and willingness to listen to our concerns.

There is one other measure that we have suggested which is still under discussion. Requiring the TOTP code to be entered on the password reset form, rather than using a secret question. We believe secret questions are bogus security, and we have an appeal to authority to back us up.

Gandi have blogged about this as well, and also given some general advice on keeping your account with them secure.

Conclusion

FastMail came out of this attack unscathed. Our domains are now even more secure, because Gandi has tons of proof on file about who we are and who our company is. Also Gandi’s processes have become more secure as a result of our experiences, so we are confident that we can safely keep our domains with them.

An important lesson learned is that just because a provider has a checkbox labelled “2 factor authentication” in their feature list, the two factors may not be protecting everything – and they may not even realise that fact themselves. Security risks always come on the unexpected paths – the “off label” uses that you didn’t think about, and the subtle interaction of multiple features which are useful and correct in isolation.

Posted in Technical. Comments Off
Follow

Get every new post delivered to your Inbox.

Join 5,258 other followers