Object storage: Cloud vs in-house

Object storage has attracted a lot of attention in recent years. Indeed, it has become the foundation data storage layer for much of the cloud and for many of the very popular services that have evolved there, from Facebook to Dropbox and pretty much everything in between.

However, it has also introduced – unnecessarily and dangerously, some would argue – an extra layer of complexity into storage planning. Its formal definition – that it is a storage architecture which manages data as objects, rather than managing it as either files or blocks – has left many people confused and wary.

In many ways, it could have been more productive and less confusing all round had the IT industry called the new concept an “enhanced file system”, or something along those lines. After all, what is an object but a song, a photo or perhaps a database? And weren't those merely files before, so why all the extra complexity?

The answer lies in the object metadata – the additional information that tells you useful things about the data. So while there is still in most cases a one-to-one mapping between objects and files, with each object also being a file, that isn't always true. More importantly, an object is not just a file, and it can be much much more than that.

It is this extra metadata that allows us to apply rules from the physical world to data in the digital world. For instance, to know how to store and move data appropriately – to replicate it, say, or to archive it to cheaper storage tiers – we need to store it by content and context, rather than simply by file name. The object store adds the metadata tags that makes that easier to do, and it adds a unique identifier to each object, eliminating duplication and allowing much greater scalability.

This scalability is one of object storage's key attractions, along with the way that object storage abstracts the file and block layers underneath, removing much of the storage administration overhead. Another is the way it stores data – which increasingly means unstructured data – in flexibly-sized containers, not as blocks of a fixed size, and stores it separately from the metadata, allowing the latter to be searched faster.

Object storage's abstraction also makes it easier to geographically distribute or replicate data, as well as providing better data protection by replacing Raid with a more advanced scheme called forward error correction or erasure coding.

There are challenges, of course, not least in that many applications were designed only to address block- or file-based storage. Several companies have developed storage gateways to bridge this gap. These present file or block interfaces to local applications, but store their data in the cloud.

Amazon and Microsoft, for example, offer LAN-based gateways that provide a local iSCSI interface to a remote object store. However, newer applications – and especially those built to run at hyperscale – are typically designed to use object storage from the outset.