Email firm runs millions of mailboxes with Scality object storage

Germany-based web mailbox provider Open Xchange has deployed around 13PB (petabytes) of Scality object storage to store more than 10 million mailboxes.

Scality’s Ring object storage has replaced OpenStack Swift in the company’s US datacentres and partially superseded a Ceph cluster in its European operations.

Open Xchange started out as a software provider, but in 2015 it merged with open source email software company Dovecot with the intention of, in part, offering hosted email services to customers.

That meant “storage became a hot topic”, said Open Xchange services vice-president, Stephan Martin.

Open Xchange’s hosted mail customers are primarily telco, cable and mobile network providers, with numbers of mailboxes per customer ranging from a few hundred thousand into more than a million. These include TalkTalk, Vodafone and Comcast.

Initially, Open Xchange deployed OpenStack Swift at its US datacentres and Ceph in Europe. But it became apparent that Swift was not well suited to the kind of storage traffic generated by email, said Martin.

“Swift is best suited to much larger files, so it was not able to keep up with the huge amount of files,” he said. “The crucial thing with email is the need to handle storage I/O [input/output] that consists of billions of small files. Swift was not suited to that. To be able to handle the I/O we needed to provide it with far more hardware – three time more than was really required – to do what we wanted to.”

Open Xchange deployed Scality Ring on 20 nodes in the US (at two sites) and 12 in Europe (also over two sites). Raw capacity is 7.6PB in the US and 5.4PB in Europe. That translates to 1.3PB and 0.9PB used in each location.

A large part of that difference is down to the replication strategy used. All mailbox content is duplicated to the other datacentre in that country. In total, four copies of all email data are retained.

Swift has been replaced by Scality storage in the US, but in Europe Ceph handles 50% of email retention.

Scality is object storage, which is very well suited to large amounts of unstructured data as is likely to be found in email infrastructure.

It stores objects with unique identifies in a flat structure, as opposed to network-attached storage (NAS), which keeps data in a tree-like file system structure and can run into performance issues when data stored runs into very large amounts of files.

The key benefits for Open Xchange have been that it can provide services to the scale it does for customers at the performance levels required, said Martin.

“The big advantage is the total saving on hardware, because Scality runs very efficiently. It’s a significant saving, although we haven’t calculated exactly how much,” he said.