Mac OS 10.9 – Infinity times your spam

UPDATE: the cause of the “infinite copy” bug was tracked down to a user side applescript. I have written a separate blog post about it.

This is a technical blog/rant. Users of FastMail who don’t use Apple’s email clients can safely skip it.

I’ve spent quite a lot of the past few days dealing with bugs caused by the Mail app in Apple’s new Mavericks update.

Apple’s mail client has been buggy with IMAP connections forever. It was infamous a couple of years ago for creating folders called INBOX.INBOX.INBOX.INBOX.INBOX.INBOX (until it hit the mailbox limit). We now block those at create time because they were causing “interesting” problems as well as being confusing.

This release has other interesting bugs that I’ve looked into in the past couple of days. When you uncheck the “keep a copy of my sent email on the server” checkbox, it auto-rechecks itself. I confirmed that report myself on our test laptop.

I also confirmed that the ‘Archive’ folder (special-use \Archive) doesn’t appear in the folder listing, and neither do any subfolders if you have any (one of my accounts does – one per year for the past *mumble* years).

So we know it’s not the breakfast of champions. That’s not a giant surprise, it’s never been the strong point of that OS. The previous revision has problems too:

229.12 UID STORE 588201 +FLAGS.SILENT ($Junk Junk JunkRecorded)
229.12 OK Completed 
230.12 UID STORE 588201 -FLAGS.SILENT ($NotJunk NotJunk)
230.12 OK Completed 
231.12 UID EXPUNGE 588201
231.12 OK Completed 
232.12 UID STORE 588201 +FLAGS.SILENT (\Deleted)
232.12 OK Completed 

Anyone who can read IMAP can see that it tries to expunge the message BEFORE the \Deleted flag is set, which is pointless. UID EXPUNGE only deletes messages with the \Deleted flag set.

http://tools.ietf.org/html/rfc4315#section-2.1

I found _that_ one because our web interface expunges all \Deleted items, so users noticed they only got the expected behaviour across multiple clients if they left the web interface open at the same time.

Millions of messages

But this doozy takes the cake. I found it nearly a week ago when we had an IO error because a user’s cache file was overflowing the 32 bit offset counter that still exists in Cyrus (I have an experiemental branch with 64 bit offsets, but it’s not ready for production yet)

They had 71 unique messages in their Junk Mail folder, but included expunged messages (we keep them for a week for backup purposes) there were over 1 MILLION separate entries in the index file. We de-duplicate on store, so the fact that there were over 100,000 copies of the most prolific message in the index didn’t totally flood the disk.

I noticed then that they were using 10.9’s mail app:

3.19 ID ("name" "Mac OS X Mail" "version" "7.0 (1816)" "os" "Mac OS X" "os-version" "10.9 (13A603)" "vendor" "Apple Inc.")

I wiped the expunged messages from the cache, emailed the user, and left it at that.

This morning I checked again, there were nearly a million messages again, so I enabled telemetry on the account and emailed the user a second time.

Here’s what I saw in the telemetry (one of many):

44.18 SELECT "INBOX.Junk Mail" (CONDSTORE)
[...]
45.18 FETCH 1:* (FLAGS UID MODSEQ) (CHANGEDSINCE 213634)
45.18 OK Completed (0.000 sec) 
46.18 IDLE
+ idling 
DONE
46.18 OK Completed 
47.18 CHECK
47.18 OK Completed 
48.18 UID COPY 3360991:3361069 "INBOX.Junk Mail"
* 158 EXISTS 
* 79 RECENT
48.18 OK [COPYUID 1318456612 3360991:3361069 3361070:3361148] Completed

Yes you read that right. It’s copying all the email from the Junk Folder back into the Junk Folder again!. This is legal IMAP, so our server proceeds to create a new copy of each message in the folder.

It then expunges the old copies of the messages, but it’s happening so often that the current UID on that folder is up to over 3 million. It was just over 2 million a few days ago when I first emailed the user to alert them to the situtation, so it’s grown by another million since.

The only way I can think this escaped QA was that they used a server which (like gmail) automatically suppresses duplicates for all their testing, because this is a massively bad problem.

I discovered my second attempt to contact the user about this problem in their Junk folder tonight. 10 times already!

Given that my colleague had just been paged by high disk usage on that user’s server – a usage which was growing fast, and which got reduced massively by expunging old deleted messages… it was time to lock the account until the faulty email client is disabled. We don’t lock user accounts lightly, but running a malfunctioning piece of software which is affecting other users and resisting attempts to contact qualifies, and a promise to disable the faulty software will be enough to resume service.

Yes, Mail.app was cleaning up after itself, but we keep deleted emails for a week, and even though it wasn’t using disk space, over a million emails were using enough meta database space that a disk had filled up. There are only 79 actual emails in this folder with a total usage of about 2MB, yet the meta files:

91M cyrus.index
906M cyrus.annotations
1.2G cyrus.cache

Over 2GB of disk usage.

Solving problems

The sad thing is – there are about 600 copies of Cyrus on our production farm, and I can roll out a new copy in about 5 minutes. There are umpteen million copies of Mail.app out there, and I can’t get them fixed on any particular schedule – so if this happens with more than one user the only real solution that I have is to code a workaround directly into our server to protect our other users.

We already wrote a special case for another one of Apple’s brilliant ideas – make the search box default to a full text search of every mailbox. The most expensive possible option for the server.

We have rate limits for other things, but we’ve never considered needing a rate limit for the COPY command – it would usually hit quota, but because these messages are expunged as fast as they are created, the quota doesn’t catch this issue.

The 4 million message 32 bit limit of the UID field will become interesting on that folder soon too, which is another thing we’ve never hit in production before. The workaround here is at least known – create a new folder, copy the messages across, delete the old folder, and rename the folder into place with a new UIDVALIDITY and new messages. – as many people have pointed out to me, it’s 4 billion, not 4 million – so much for last minute ideas when writing late at night. Sorry for the confusion. It would take a lot longer then a few days to hit the limit!

Posted in Technical. Comments Off

Faster than native, introducing FastMail’s new mobile web interface

For the last few months we’ve been beta testing a new mobile user interface. We feel it’s now ready for general use and have just rolled it out to production.

To access the new interface, just go to https://www.fastmail.fm in a web browser on your phone/mobile device and log in to your FastMail account. If your phone supports it (iOS 6+, Android 4+, Windows Phone coming soon), you’ll automatically get the new interface.

This new interface is built on the same underlying technology as our current desktop interface, and thus includes all the advanced features of that interface including instant actions, conversations and fast cross-folder searching.

We’ve also worked hard to make the new interface feel perfectly natural as a finger driven mobile interface. We’ve placed tap targets near where fingers are likely to be and added swipe actions to allow quick archiving or deleting of emails. Simple transitions make it clear where you are in the interface at any time making it easy to navigate.

But most importantly, the new mobile interface is fast! We’ve gone to great lengths to reduce the number of round trip requests between your phone and our servers, making the interface load and feel fast even over the high latency connections of mobile networks. In many cases, it is faster than a dedicated email app.

We’re pushing the limits of what’s currently possible with a web based application and think the results speak for themselves. Below we’ve created a short video showing a few key features. Take a look, then try it out by logging in or signing up at https://www.fastmail.fm.

Posted in News. Comments Off

FastMail’s servers are in the US: what this means for you

As you may be aware, there have been many reports recently of secret data mining programs conducted by the US government. These reports have included mention of secret network interception and “backdoor” access to private email accounts. While we have no position on the veracity of these claims, we have had many queries about what, if anything, this might mean for us and for our customers, given that our primary servers are located in the US. This is our response to those questions.

As noted in our recently updated privacy policy, we are an Australian company subject to Australian law. We are required to disclose information about specific individual accounts to properly authorised Australian law enforcement with the appropriate supporting documentation, which means a warrant signed by an Australian judge. We do not co-operate with any kind of blanket surveillance, monitoring or “fishing expeditions”, and we do not give out user information to anyone outside Australia.

There are some rare cases of overseas entities applying to Australian authorities for information under a Mutual Assistance treaty. In that case proper Australian documentation must be issued before we will do anything. These cases are particularly rare because the burden of proof required is very high, and the chain of events is very long (ultimately, each case currently requires sign off from the Australian Attorney General). The Mutual Assistance treaties also have (amongst other things) a test of whether the putative offense would be illegal in Australia, not just in their country of origin.

Australia does not have any equivalent to the US National Security Letter, so we cannot be forced to do something without being allowed to disclose it.

It has been pointed out to us that since we have our servers in the US, we are under US jurisdiction. We do not believe this to be the case. We do not have a legal presence in the US, no company incorporated in the US, no staff in the US, and no one in the US with login access to any servers located in the US. Even if a US court were to serve us with a court order, subpoena or other instruction to hand over user data, Australian communications and privacy law explicitly forbids us from doing so.

It might be possible for the US government to lean on the Australian government or other international legal body to compel us to hand over data but this likely to be an expensive, time-consuming and highly visible process. In our opinion those barriers make it extremely unlikely to happen.

There are of course other avenues available to obtain your data. Our colocation providers could be compelled to give physical access to our servers. Network capturing devices could be installed. And in the worst case an attacker could simply force their way into the datacentre and physically remove our servers.

These are not things we can protect against directly but again, we can make it extremely difficult for these things to occur by using strong encryption and careful systems monitoring. Were anything like this ever to happen we would be talking about it very publically. Such an action would not remain secret for long.

Ultimately though, our opinion is that these kinds of attacks are no different to any other hacking attempt. We can and will do everything in our power to make getting unauthorised access to your data as difficult and expensive as possible, but no online service provider can guarantee that it will never happen.

As a customer, you need to weigh the benefits of keeping your data with us against the risks of that data being disclosed. You may wish to obtain private legal advice to help you assess that risk. And if you come to the conclusion that keeping your data with us is too risky, then we respect that (though please do tell us why!)

If you have any further privacy concerns we haven’t addressed, please email privacy@fastmail.fm.

Posted in Uncategorized. Comments Off

Updated privacy policy

After the recent sale of FastMail back to the developers, we decided it was a good time to review and update our privacy policy. We hope this makes it clear that we strongly value our users privacy and will continue to do so in the future.

The new policy is available at https://www.fastmail.fm/help/overview_privacy.html and is included below.

The FastMail Team


Privacy Policy

At FastMail, we take the privacy of our users very seriously. We want to make our policies on managing your data clear and understandable, so we’ve tried to write our privacy policy in plain English. If you have any further privacy concerns we haven’t addressed, please email privacy@fastmail.fm.

Jurisdiction

FastMail is an Australian company and as such is subject to Australian law. Australia has strong privacy laws in relation to email, specified in the Telecommunications (Interception and Access) Act 1979. The Electronic Frontiers Australia organisation has an excellent summary; this privacy policy tries to make it clear how it applies in practice to FastMail.

Surveillance and law enforcement

We do not participate in, or co-operate with, any kind of blanket surveillance or monitoring. (We also point out that Australia does not have any equivalent to the US National Security Letter, so we cannot be forced to do something without being allowed to disclose it.)

We also take technical measures where feasible to prevent surveillance of our users occurring without our co-operation, such as:

  • using encrypted SMTP for sending your mail when the receiving server supports it.
  • mandating encrypted access for webmail, IMAP and POP.
  • using Perfect Forward Secrecy where possible for all encrypted connections.
  • encrypting communications between our data centres.

Like any company, we can never guarantee our measures are 100% effective, as we don’t know the full capabilities of any attackers. However, these measures do act to increase the difficulty and expense of any surveillance.

As an Australian company, we are required to disclose information about specific individual accounts to properly authorised Australian law enforcement with the appropriate supporting documentation. This means we need to see a warrant signed by an Australian judge before we will hand over any email data. Such requests must always be for specific accounts; we do not participate in or co-operate with "fishing expeditions". As a guideline, in the last year we disclosed information on fewer than 50 accounts.

We do not directly disclose any information about our users to law enforcement from outside Australia, and indeed our understanding of Australian law is that it would be illegal for us to do so.

Overseas law enforcement may apply via an appropriate mutual assistance treaty to obtain information on our users. If the request is approved, then Australian documentation will be issued for disclosure of this information.

This distinction may seem academic, but in our experience the extra administrative overhead, and the additional layers of judicial oversight mean that we receive very few valid requests that originate from overseas and they must always be targeted at specific accounts.

We do not condone illegal activity. We deal with all law enforcement requests personally and we are satisfied that all we have seen are justified.

Data mining and profiling

We do not sell or give information about our users to any third parties. Payments are securely handled via Pin, Global Collect or PayPal; your credit card details are never transmitted to our servers. Pin and/or Global Collect store your credit card details and address for the purpose of future payments with FastMail, unless you have requested your payment details not to be stored. Pin’s privacy policy is available at https://pin.net.au/privacy. Global Collect’s privacy policy is available at http://www.globalcollect.com/Privacy-statement/. PayPal’s privacy policy varies depending on your country of residence; you can select your country to find the relevant privacy policy at https://www.paypal.com/webapps/mpp/ua/legalhub-full.

Incoming messages are scanned for the purpose of spam detection unless you disable spam protection for your account. We may also scan some outgoing messages with the same software to prevent people using our service to send spam. Emails you report as spam are automatically analysed to help train our spam filter. Also, if enabled, emails reported as spam are forwarded on to some external email reporting services. These services aim to help monitor and reduce overall spam on the Internet. Currently the services we report to are Return Path and LashBack. These may change in the future. If you don’t want this, you can disable the reporting in the FastMail advanced settings.

To make message searching fast, we build an index of your messages (this is a table, just like you would find at the back of a reference book, in which you can look up a word to quickly find the emails in which it appears).

No information from any of these activities is used for any other purpose, or to compile any kind of profile on our users.

Data retention

We retain backups of deleted messages for at least a week. This is for the purpose of restoring messages in case of accidental deletion. After this point, deleted messages will be purged from all our backups, although the time this takes to happen may vary due to automated load balancing.

We normally keep logs of email and server activity for up to 6 months. This is for the purposes of diagnosing and fixing problems, which are often reported to us weeks or months after they occur. Message subjects may be contained in these logs, but not message bodies. Aggregate or anonymous data, which cannot be linked to individual user accounts, may be kept for longer periods, for the purpose of improving the FastMail service.

Backups and logs may be kept longer than these limits in special circumstances. For example, if a problem is taking a long time to resolve, logs relevant to that investigation may be retained. Or if a server that contains backups or logs is temporarily offline because of a fault, then those backups or logs may not be deleted until the server is brought back up.

These situations are unusual, however, and when they do occur, they are temporary.

Account deletion

Should you close your account, all data will be permanently deleted 7 days after closing. It may take a further 2 weeks to purge from all our backups.

Posted in News. Comments Off

Exciting news: FastMail staff purchase the business from Opera

In 2010, FastMail was bought by Opera Software. The developers and staff of FastMail have now bought back the company. This means that FastMail is once again an independent company, dedicated to building the best possible email experience for our users. We have big plans for the future, and we will continue to roll out new features and enhancements over the coming months.

There are no configuration changes or any other changes you need to make. All existing accounts will continue to run as they do now with the same billing cycle, pricing, features, reliability, security, etc.

In case you have any questions, we’ve tried to address the main issues below.

  • Why has Opera sold you? Are you in trouble?

    Not at all. Opera has undergone an internal change of strategic direction and an email service no longer fits within their long term vision. With Opera’s investment in development and infrastructure over the last 3 years, FastMail has continued to increase its rate of growth and profitability. We came to the mutual conclusion that FastMail’s future would be better served as a separate company.

  • How will this affect future development work?

    FastMail is keeping all existing FastMail related staff. We believe we have all the resources and talent needed to keep developing and growing FastMail now and going forward into the future.

  • What sort of things do you have planned?

    A hugely improved mobile interface, CardDAV support to allow synchronisation of contacts between devices, a calendaring service including CalDAV support for synchronisation of events between devices, improved backend and searching performance. All these things are currently in active development and slated for release within the next year.

  • This all sounds great. Is there any way I can help?

    The best way to help us is continue to use FastMail. It’s the support we get from our users that allows us to keep running and developing the service.

    Tell your friends that there’s a real alternative to the big corporations. One that doesn’t show ads, respects your privacy, and is fully committed to keeping the service going forward.

    Tweet about us. Post about us on your blog. Make you and your boss happy by switching your work email to FastMail :)

  • How does this affect the privacy of my email and other data?

    We have always taken our users’ privacy very seriously and this will not change. We’re working on publishing an updated privacy policy next week that will explain in clear wording exactly how your data is treated. We’ll post to the blog with more information soon.

Thanks for using FastMail. We’ve put a lot of thought and effort into building the fastest, easiest and most powerful way to access your email. We look forward to providing you with the best service we can.
 
The FastMail Team

Posted in News. Comments Off

iOS 7 Mail App uses multi-folder body searches by default

This is a technical post. Regular FastMail users subscribed to receive email updates from the FastMail blog can just ignore this post.

We’ve recently been testing out the Mail application in iOS 7 and found that there’s been a couple of significant changes, especially when you search your email. If you run an IMAP server with users that connect using the iOS Mail application, you might be interested in what we found. The two main ones are:

  1. Rather than searching the current folder, by default, it searches all folders. To do that, it opens many IMAP connections at once for searching each folder concurrently. We’re not sure on the upper bound on the number of connections it will make, but saw at least 10 in one case.
  2. Rather than searching the Subject/To/Cc/From fields, by default, it searches all those fields as well as the message body.

Both of these changes are actually great for the user experience, but they create potentially large headaches for IMAP server maintainers.

Clients making multiple IMAP connections at once isn’t new, but the number of potential IMAP connections an iOS client might now make is large. Some administrators limit the total number of IMAP connections a user can make at once. They might have to raise this or iOS might start returning unexpected results.

The bigger concern is the body searching. The search string iOS now sends is:

tag UID SEARCH RETURN (ALL) (OR FROM "Foo" (OR SUBJECT "Foo" (OR TO "Foo" (OR CC "Foo" BODY "Foo")))) NOT DELETED

IMAP search semantics suggest that a body search is supposed to be a pure sub-string search (BODY <string>: Messages that contain the specified string in the body of the message). Depending on your IMAP server, your message bodies may or may not be indexed in a way that allows sub-string searching. If they’re not, then a BODY search is potentially very expensive, requiring every message to be opened and searched individually. Doing that simultaneously across many folders might generate a sudden and significant IO load on a server.

Our plan at FastMail is to detect iOS clients, and convert all searches into FUZZY searches. This causes matches to be done on “terms” rather than pure sub-strings, but allows us to use our xapian powered index which should make matching and fetching results much, much quicker.

Posted in Technical. Comments Off

Fastmail uses perfect forward secrecy with https/TLS connections

There’s been a number of articles recently about perfect forward secrecy (PFS). The main aim of PFS is to ensure that even if the private SSL/TLS key for www.fastmail.fm was ever compromised, it would still be impossible to decrypt any existing captured traffic between users and our server. If you’re looking for more information, the linked articles above are worth reading to get a better overview. For PFS to work, both the server (us) and the client (your web browser) must support it.

Fastmail has supported PFS via ECDHE for some time now (since July 2012). Unfortunately a few browsers don’t support ECDHE.

Today we’ve updated our ciphers to the best practice recommended by SSL Labs. Using the SSL Labs site tester on www.fastmail.fm shows that we now support PFS on all major browsers except for IE 8 on Windows XP, which has no support for PFS and so can never support it.

We’re pretty sure that this change won’t have any compatibility issues with old clients (which should fall back to older ciphers), but we’ll keep an eye out in case there’s any reported problems.

Posted in News, Technical. Comments Off
Follow

Get every new post delivered to your Inbox.

Join 5,427 other followers