NVMe-over-TCP brings super-fast flash over standard IP networks

Recap: The NVMe advantage 

The NVMe specification was initially created to provide low-latency, high-bandwidth access to robust flash storage. It has several key advantages over legacy protocols such as SAS and SATA.

Most notably, it provides multiple channels between the CPU and flash drive, which is a huge advance on the bottlenecks caused by the use of SCSI at the heart of SATA and SAS. 

Version 1.0 of the NVMe protocol was created in 2008, when flash-based storage was starting to replace mechanical disk-based drives, which were slower and far more fault-prone. At the time, flash storage connected to computers predominantly via SATA or SAS. The storage industry quickly realised that this hampered the overall value proposition of flash. 

To remedy this, the NVMe spec was created. The biggest advantage over the legacy protocols was that NVMe connected over the PCI Express (PCIe) bus. 

PCIe provides a high-speed, high-bandwidth connection between the CPU and components such as graphics cards and flash storage. On an architectural level, it provides multiple I/O lanes for data transfer and has a sophisticated queueing system that softens the performance degradation that comes with periods of high-intensity I/O demand.  

Unlike older serial connections, the PCIe protocol acts a bit like a network, routing packets between components and performing a sophisticated form of traffic management. Also, PCIe drives share a common bus, allowing components to communicate directly with each other, which reduces latency when copying between drives. 

Why TCP?

So, although it is possible to connect NVMe flash via PCIe, and over wider distances to connect storage and hosts via the various NVMe-over-fabrics methods such as RDMA/Ethernet and Fibre Channel, the use of TCP (Transport Control Protocol) provides a potentially cheaper, more flexible alternative.

TCP actually dates back to the late 1970s, and its solid design principles have ensured its relevance nearly 50 years later. On a fundamental level, TCP is a fault-tolerant protocol for sending data across a network. Above all, it emphasises accuracy over naked speed. 

If a packet gets lost or degraded, the computer’s TCP implementation will request its retransmission. And should network congestion cause packets to arrive in the incorrect order, TCP can rearrange them. 

TCP is one of the cornerstones of the modern internet, but its cautious, conservative attributes make it well suited for the datacentre world, where accuracy and reliability are key. 

NVMe is designed with performance in mind, while TCP focuses on accuracy and reliability. Within the context of NVMe-over-TCP, the latter protocol is responsible for facilitating the communication between a remote client and the NVMe controller. It establishes connections, defines queues and encapsulates data.

Business base for NVMe-over-TCP

NVMe-over-TCP satisfies many key storage needs for data-intensive corporate users. The inherent rapidity of NVMe storage provides fast, low-latency access to robust flash storage that mirrors the performance of traditional direct-attached storage.

But where NVMe-over-TCP really excels is in deployment, with lower initial costs and setup times than competing NVMe networked schemas. 

NVMe-over-TCP does not require any additional extensions, hardware or software to work. Instead, it uses the ubiquitous TCP protocol, which your existing infrastructure (for example, switches and NICs) is guaranteed to support. This means the initial capital costs are limited to the drives and controllers, as well as the human time required to implement them within the context of an existing network. 

This simplicity means there is minimal ongoing human capital costs when maintaining NVMe-over-TCP. You won’t need to hire a consultant with experience in a specific niche networking appliance, or dedicate vast human hours to managing the system. 

In fact, the biggest Achilles’ heel of NVMe-over-TCP is, arguably, its inherent newness.

Consequently, there isn’t much of an ecosystem surrounding the technology, which limits access to third-party support should you require it. Similarly, the technology lacks the veritable forest of documentation, blog posts and tutorials that surround legacy technologies. So the biggest risk surrounding NVMe-over-TCP is that those who choose to embrace it effectively do as early adopters – at least for now. 

NVMe-over-TCP products

Israeli startup Lightbits is heavily backed by big storage players. It offers an array-type product based on NVMe-over-TCP using its LightOS in a LightBox hardware to provide advanced storage management features – thin provisioning, compression, striping, data protection, and so on.

Toshiba offers its KumoScale NVMe-over-fabrics array software, which allows for transport over TCP as well as RoCE.

Vendors including Mellanox, Kalray and Marvell have adapter products that support NVMe-over-TCP.