Western Digital aims to help rethink unstructured data storage

Drive giant Western Digital (WD) has launched an open source initiative to allow the development of software-defined storage aimed at storing large volumes of relatively unchanging unstructured data, such as video, at costs significantly lower than today.

The architectural underpinning to this “zettabyte-scale” storage will major around high capacity but less performant media that are dependent on sequential writes, such as shingled magnetic recording (SMR) hard disk drives and so-called Zoned Name Space (ZNS) NVMe flash media.

The key driver for Western Digital is the rise of workloads based around large quantities of unstructured data that is written once and accessed many times after that. The company mostly has video content in mind but also believes big data, internet of things (IoT) content will fit this category.

In that, it contrasts this approach to the world of random input/output (I/O) and read/writes of most contemporary storage architectures.

In the software-defined storage initiative – called Zonedstorage.io – a key process is to allow serialisation of I/O higher in the stack so that capacity in SMR and ZNS drives can be used.

The recording tracks in SMR HDDs are overlapped so that once written they are difficult or impossible to re-write. But the efficiencies gained in areal density mean, for example, that a single SMR hard drive can hold around 15TB currently, with 20TB on the near horizon.

Meanwhile, ZNS NVMe flash offers similar density gains. ZNS does away with much of the redundant capacity and Dram needed to run conventional enterprise flash. The flipside of that is that data needs to be written to its zoned storage grid more carefully, i.e sequentially, so it is best suited to the kind of workloads Western Digital proposes.

The main storage chip makers – including Western Digital – have ZNS NVMe in development with GA planned for later this year and into 2020.

Zonedstorage.io will be a Linux-based development environment for file systems, object storage and associated storage libraries and toolkits.

Western Digital CTO, Martin Fink, said: “The demand for this is all from customer pull. They want to move intelligence to the host for simpler and highly optimised devices. The customer will benefit by gaining far more storage at much lower costs.”

For 1ZB of storage, Western Digital suggests 25 million fewer hard drives would be needed if SMR was used in place of conventional disk.