Disaggregated systems try to unlock the NVMe conundrum

Removing the controller bottleneck

The answer so far to the NVMe performance conundrum has been to remove the bottleneck completely.

Rather than have all I/O pass through a central controller, why not have the client system access the drives directly?

With a fast, low-latency network and direct connectivity to each drive in a system, the overhead of going through shared controllers is eliminated and the full value of NVMe can be realised.

This is exactly what new products coming to the market aim to do. Client servers running application code talk directly to a shelf of NVMe drives, with the result that much lower latency and much higher performance numbers are achieved than with traditional shared systems.

NVMe implementation

Creating a disaggregated system requires separation of data and control planes. Centralised storage implements the control and data path in the controller. Disaggregated systems move control to separate elements of the infrastructure and/or to the client itself.

The splitting of functionality has the benefit of removing controller overhead from I/O. But there is also a negative effect on management, as the functions that were performed centrally still have to be done somewhere.

To understand what we mean by this, imagine the I/O that occurs to and from a single logical LUN on shared storage mapped to a client server. I/O to that LUN is done using logical block address (LBA).

The client writes from block 0 to the highest block number available, based on block size and capacity of the LUN. The controllers in the storage take the responsibility of mapping that logical address to a physical location on storage media.

Then as data passes through a shared controller, the data block will be deduplicated, compressed, protected (by Raid or erasure coding, for example) and assigned one or more physical locations on storage. If a drive fails, the controller rebuilds the lost data. If a new LUN is created, the controller reserves out space in metadata and physically on disk/flash as the LUN is used.

In disaggregated systems, these functions still need to be done and are, in general, passed out to the client to perform. The client servers need to have visibility of the metadata and data and have a way to coordinate between each other to ensure things go smoothly and no data corruption occurs.

Why disaggregate?

The introduction of NVMe offers great performance improvements.

In certain applications low latency is essential, but without disaggregation the only real way to implement a low-latency application is to deploy storage into the client server itself. NVMe flash drives can deliver super low latency, with NVMe Optane drives from Intel giving even better performance.

Unfortunately, putting storage back into servers isn’t scalable or cost effective and was the original reason shared storage was first implemented. Disaggregation provides a middle ground that takes the benefit of media consolidation and (seemingly) local storage to get the highest performance possible from new media.

The type of applications that need low latency include financial trading, real-time analytics processing and large databases where transaction performance is a direct function of individual I/O latency times.

There’s an analogy here to the early days of flash storage, where all-flash arrays were deployed in the enterprise onto applications that would be expensive to rewrite or simply couldn’t be speeded up by any other method than delivering lower latency.

In the first implementations it’s likely we will see disaggregated systems deployed on only those applications that will benefit most, as there are some disadvantages to the architecture.

Compromises

As highlighted already, depending on the implementation, client servers in disaggregated systems have a lot more work to do to maintain metadata and perform calculations such as Raid/erasure coding, compression and deduplication.

Support is limited to specific operating systems and may require the deployment of kernel-level drivers or other code that creates dependencies on the OS and/or application. Most systems use high-performance networks such as InfiniBand or 40Gb Ethernet with custom NICs.

This increases the cost of systems and will introduce support challenges if this technology is new to the enterprise organisation. As with any technology, the enterprise will have to decide whether the benefits of disaggregation outweigh the support and cost issues.

One other area not yet fully determined are the standards by which systems will operate. NVMe over a network or NVMe over Fabrics (NVMeF) is defined by the NVM Express organisation, and covers the use of physical transports such as Ethernet and Infiniband with access protocols such as RDMA over Converged Ethernet (RoCE) and Internet Wide-Area RDMA Protocol (iWarp), which provide remote direct memory access (RDMA) from client server to individual drives.

Some suppliers in our roundup have pushed ahead with their own implementations in advance of any standards being ratified.

NVMe supplier systems

DVX is a disaggregated storage system from startup Datrium. The company defines its offering as open convergence and has a model that uses shared storage and DRAM or flash cache in each client server. The company claims some impressive performance figures, achieving an IOmark-VM score of 8,000 using 10 Datrium data nodes and 60 client servers.

E8 Storage offers dual or single appliance models. The E8-D24 dual controller appliance offers Raid-6 protection across 24 drives, whereas the E8-S10 implements Raid-5 across 10 drives. Both systems use up to 100GbE with RoCE and can deliver up to 10 million IOPS with 40GBps throughput. E8 also offers software-only systems for customers that want to use their own hardware. Note that the dual controller implementation is to provide metadata redundancy.

Apeiron Data Systems offers a scale-out system based on 24-drive NVMe disk shelves. Client servers are connected using 40Gb Ethernet. Apeiron claims performance figures of 18 million IOPS per shelf/array and an aggregate of 142 million with eight shelves. Latency figures are as low as 100µs with MLC flash and 12µs with Intel Optane drives.

Excelero offers a platform called NVMesh that is deployed as a software system across multiple client servers. Each client server can contribute and consume storage in a mesh architecture that uses Ethernet or Infinband and a proprietary protocol called RDDA. Systems can be deployed in disaggregated mode with dedicated storage or as a converged system. Performance is rated as low as 200µs, with 5 million IOPS and 24GB/s of bandwidth.