Storage analytics: How AI helps storage management

Hardware vendor tools versus multi-supplier monitoring

Most – but not all – storage analytics tools come from the hardware suppliers, giving IT managers a useful snapshot on the health of their arrays or storage subsystems. But this only gives a limited picture of a complete system or stack, especially when more than one vendor is involved.

“Most storage managers will use the default tools that come with the equipment,” says Andy Buss, at analyst house IDC. “Enterprises have the desire to run centralised tools, but often end up not doing so. They revert to what comes with their equipment.”

More multi-supplier storage monitoring and analysis tools are coming onto the market, partly spurred by a need for firms to manage hybrid environments and partly by the growing use of standardised  application programming interfaces (APIs) – especially the representational state transfer (REST) API – for storage.

But using multi-supplier tools involves a trade-off between system-wide visibility and detailed feedback on system performance. Supplier-independent tools do not, as yet, gather sufficient data to optimise every manufacturer’s equipment fully.

“A lot of the new management tools are more multi-vendor capable,” says Buss. “You get a base level of functionality. You don’t get all the bells and whistles, but you do have more control over your entire infrastructure.”

There is growing appetite among IT teams, he says, for a “single pane of glass” to manage infrastructure, including storage. Such systems do come at an additional cost, and companies have been reluctant to pay for them.

But this is changing, in part due to their experience in operating systems such as Azure Stack and Amazon Web Services (AWS) Outposts, which come with sophisticated management built in.

Predictive tools and AIOps

Storage and system analysis tools are also becoming smarter. Hardware and management tool suppliers are turning to advanced analytics, and even artificial intelligence (AI) and deep learning, to improve system performance.

This can involve moving datasets to the most cost-effective storage tier, moving files away from a sub-system that shows signs it might fail, or consolidating data to maximise the balance between capacity utilisation and performance.

Companies are turning to these tools to cope with larger and more complex environments, including those that mix cloud and on-premise resources. AI and machine learning (ML) are increasingly seen as one way – sometimes the only way – to cope with that complexity and deliver IT performance to the business.

According to recent research by analysts ESG, some 23% of companies see AI and ML for system management as a top priority for datacentre modernisation.

“These intelligence features are vital for any IT environment of any significant size,” says ESG’s Scott Sinclair. “Organisations do not have the excess people to spending time continuously optimising every environment as workload usage evolves or to diagnose complex component failures.” Instead, they are turning to systems to do so.

Gartner has called this “AIOps, which stands for “artificial intelligence for IT operations”.

Gartner predicts that by the end of 2025, 40% of new deployments of infrastructure products, including storage and hyper-converged systems, will be AIOps-enabled, up from less than 10% in 2020.

The new tools proactively analyse capacity and performance status, predict potential issues that may cause data services disruption, and provide actionable advice to resolve Level 1 problems enhances storage utilisation efficiency. 

Julia Palmer, Gartner vice-president, says: “Storage tools have always delivered on some metrics for capacity and performance, but it was not good enough as it required someone with storage expertise to constantly monitor it. AIOps tools, however, look for anomalies, patterns of consumption and performance trends and correlate it with normal behaviour of the specific customer system and other systems supported by the vendor.”

What’s available?

By bringing together richer data sources and a degree of artificial intelligence, some suppliers claim significant improvements in system performance and availability.

HPE’s InfoSight, for example, is viewed as one of the most advanced. It monitors 100,000 systems and between 30 million and 70 million sensors worldwide for maintenance and performance issues.

InfoSight claims to detect and fix 86% of potential problems without the need for human intervention. As many as 54% of problems picked up by InfoSight are “outside” storage and somewhere else in the stack, the company says. For now, InfoSight only works with HPE Nimble systems, and 3PAR technology.

Virtana – formally Virtual Instruments – is also a leader in system performance management, with a focus on hybrid architecture as well as private cloud.

There are other suppliers in the market too. IBM’s cloud-based Storage Insights, NetApp’s Active HQ and Hitachi Vantara are some of the best known.

Microsoft also has extensive monitoring capabilities in Azure, via REST APIs, as well as through Windows Server, where that platform is used to run storage-dense server hardware. VMWare also has its toolsets for virtualised environments, through its intelligent optimisation tools for vSAN.

Together these applications give CIOs powerful tools to monitor and optimise their environments, as well as prevent failures.

Smarter systems, smarter storage

The industry is, however, still at an early stage on its journey to intelligent, device-independent storage management.

Areas of growth are likely to include more support for hybrid environments, more granular management of the different types of flash storage on the market, and potentially support for capacity-based pricing. Tools such as Virtana and vSAN can already take account of data ingress and egress costs.

“The predictive capabilities in some of these systems is really amazing,” says ESG’s Sinclair. “Intelligent systems typically offer recommendations to optimise system performance or capacity, while some even offer the option to self-optimise.”

“In a similar fashion, these systems can also often automatically diagnose issues and recommend actions. It’s difficult to make claims on predictive capabilities, because the results can change based on the environment. But some vendors rely on these capabilities to make higher level claims, such as 100% availability guaranteed for example.”

Whether such claims can be substantiated remains to be seen. It will be easier for suppliers to hit reliability targets in largely monolithic, single vendor environments than for more complex systems.

And reliability and performance metrics will vary depending on the workload and applications. A system that prioritises short-term performance over reliability might not be giving the business what it needs.

As a result, storage analytics will continue to work alongside human analysts, suggests IDC’s Andy Buss. “Systems need to be sustainably reliable,” he says. “For the technology to be accepted, it needs to work as an assistant. Few companies will turn over their IT or storage infrastructure entirely to AI.”