idspopd - Fotolia
PCIe SSD: Cache vs storage tier
Adding PCIe flash is a popular way of speeding up application response times, with more rapid I/O guaranteed, but should PCIe SSD be added as cache or as a tier of storage?
Flash storage provides significantly more performance (IOPS and throughput) and lower latency than hard disk-based media.
Of course, flash comes with a higher price tag – up to 10 times more than large-capacity hard drives. This makes it impractical for many IT organisations to consider moving all their data to flash technology.
As a compromise, flash can be used to accelerate active I/O, either by using it as a cache or as a tier of storage comprising a percentage of overall storage capacity.
But caching and tiering are two very different approaches that target fast storage media at active data. A cache is essentially a copy of active data that sits between the application and back-end storage, such as cheap disk drives. In contrast, storage tiering places data on multiple types of media (faster, more expensive versus cheaper, slower) depending on the activity of the data itself.
The use of DRAM or system memory for cache has been a feature of computer architectures since the mainframe. And in storage, we have been using cache to accelerate I/O in external arrays for the past 25 years.
Today we can choose to cache active data within the storage array, within the hypervisor, within the operating system (OS) or within the application itself. From a storage perspective, caching in the hypervisor or OS reduces the I/O that goes to external media. Meanwhile, placing cache in the array reduces the amount of times array software needs to go to slower disk-based media to read or write data.
There are three types of caching: write-back, where I/O is immediately confirmed to the host after writing into the cache; write through, where writes are written to the cache and external media; and write-around, where the cache is bypassed for write I/O.
In all instances, the cache accelerates reads by servicing active read requests directly from the cache itself. Use cases for each caching technique depend on the I/O profile of data from the application.
In many cases, write-through caching (which accelerates only reads) will be good enough. Write-back allows write I/O to be accelerated, but needs to be implemented in a fault-tolerant way to protect against hardware failures. Write-around works well with large sequential I/O that might not immediately be re-read by the application.
Tiering deep dive
Tiering is the process of placing data onto multiple types of media, depending on the cost/performance characteristics of the data. Very active data justifies being on faster, more expensive media, while relatively inactive data is best suited to large-capacity hard drives. As the I/O profile of data changes, it can be moved between tiers to ensure more expensive media is used efficiently.
The process of moving data between tiers is typically a reactive process, based on looking at the activity of data over time.
Initial implementations placed entire logical volumes or LUNs on a single tier, which could be inefficient with large volumes – imagine a 2TB volume with only a few hundred gigabytes of active data.
Over time, suppliers improved the tiering process, introducing sub-LUN tiering (where a logical volume is made up of storage from more than one tier) and dynamic tiering, where the storage array moves data around automatically based on predefined policies.
PCIe flash
NAND flash comes packaged in a number of forms, including support for the PCIe protocol. Physically, this can mean AIC (add-in-card) devices or the more traditional 2.5in drives.
NVMe is emerging as the new protocol of choice for writing to PCIe-based storage because it offers advantages, such as greater parallelism (more I/O queues, longer queues), that suit flash media.
In terms of capacity, PCIe flash devices range from a few hundred gigabytes up to multiple terabytes in capacity. The capability of PCIe devices is also widening, with lower-endurance and lower-cost products appearing based on TLC and 3D-NAND technology.
PCIe for cache or storage
PCIe devices are great to use as cache or a storage tier. The performance of applications can be increased greatly as PCIe devices typically have read/write access times measured in tens of microseconds, compared with low milliseconds for spinning media.
Using PCIe flash significantly reduces I/O latency for applications and this translates directly into faster application response times or the ability to process more workload.
Obviously, the decision to use flash as a cache or tier depends on a number of factors, including the capabilities of the device itself.
Caching tends to be more responsive to application workload profile changes as active data changes over time. Caching algorithms simply evict inactive data in place of the application workload that has become busy.
Tiering algorithms
In contrast, tiering algorithms examine data over time and make decisions to move data between tiers. There is a trade-off between the cost/overhead of moving data and the potential savings that can be achieved, so data might be moved hourly or daily as a result.
Caching does have one major issue – the cache can be overwhelmed when the amount of active data exceeds the cache capacity. In this instance, data has to be retrieved from the slow media, resulting in a loss of performance.
Read more on PCIe SSD
- The PCIe SSD market is still important, but PCIe flash drive makers are split between those that continue to develop their products and those for whom product evolution has stalled.
- We run the rule over PCIe SSD and assess its potential as a server-side alternative to all-flash storage, as cache in conjunction with shared storage and in hyperscale environments.
With tiering, all active data sits on the fast tier, so this problem will not arise. Also, “cache misses” (when active data is not already in cache) can result in poor I/O performance, whereas having all the data on a flash tier guarantees I/O performance for that data.
This makes caching more suitable for workloads that can tolerate some spikes in performance/latency and makes tiering more suitable when those issues cannot be tolerated.
Suitable PCIe devices
Suppliers in the market today sell a range of products that suit either tiering or caching. Read-intensive workloads will be more suited to devices with lower endurance capabilities because flash endurance is dictated directly by the amount of times that data is written to the device.
For example, SanDisk Fusion products (formerly Fusion-io) come in two ranges (SX and PX series), suited to read-intensive and more general workloads, respectively. This is reflected in the endurance ranges of the two products, with SX rated as 4-22 PBW (petabytes written) and PX from 12-64. PBW is the amount of data the supplier will guarantee can be written to the device.
Another feature that must be considered is data protection. Cached data will always also exist on shared storage, so Raid-style protection for the cache is not required. The only exception to this is write-back caching, where I/O is acknowledged to the host before being written to storage. Here, data is usually mirrored to another device.
With tiering, the data exists only on the tier and RAID or mirroring is essential to protect against device failure. Suppliers such as Micron, with its new 7100 and 9100 series products, provide power-loss protection to ensure data is not lost or corrupted in the event of an unexpected power failure.
As product costs reduce, the use of cheaper TLC NAND-based products will become more prevalent and so more attractive for a wider range of data. Samsung has brought products to the market using 3D and TLC technology, such as the PM1725 and PM953. With capacities up to 6.4TB, the PM1725 may be more suited to being a tier of storage in its own right, or may have to be placed in front of a significant amount of traditional media.
More details on suppliers and products can be found in our recent roundup of the PCIe flash market.
