In this buyer’s guide, we look at the challenges surrounding why enterprises are struggling to deal with data, the use of cloud data management platforms, artificial intelligence, and recommend some best practices
In 2018, two industry experts forecast that the world’s collective data growth would be staggering. Research firm IDC predicted that data would rise from 33 zettabytes (ZB) in 2018 to 175ZB by 2025, with a compound annual growth rate (CAGR) of 61%.
Similarly, networking supplier Cisco Systems anticipated that global IP traffic would rise to 4.8ZB a year by 2022 – more than three times the 2017 rate. This rise was to be driven by the increased use of internet of things (IoT) device traffic, video, and the sheer number of new users coming onboard.
But although predictions are insightful to a point, the more worrying trend is that many enterprises are struggling with the pain of legacy data management, endless backup jobs and myriad hardware and software solutions for managing and protecting data.
Exacerbating this is the pressure for enterprises to adopt cloud computing because of remote working and complex cyber security challenges such as ransomware attacks.
According to Darian Bird, principal adviser for Ecosystm, the two main challenges enterprises face with data management are data integration and data quality.
“Our recent research finds that 77% of organisations globally find data integration a key challenge for data management because of the diversity of data types, sources and environments,” he says.
Closely connected to data integration is the need to ensure data quality as organisations look for accurate information to capitalise on actionable intelligence, says Bird.
Ravi Rajendran, vice-president for Asia Pacific at Cohesity, agrees, adding that as enterprise data has expanded so rapidly, legacy systems are unable to keep up because data is all over the physical and virtual enterprise.
“By not having a centralised, consistent view of data and easy accessibility to data by any and all of their applications that require it, they will never be able to maximise the value of their data,” he says.
“In most firms, IT attempts to manage each silo through different vendors, with proprietary systems and user interfaces, which is ultimately challenging, expensive and highly inefficient.”
Rajendran adds that many enterprises don’t know what kind of data they have, where it resides and who has access to it – all of which will lead to a serious threat in data security and risk management.
“The combination of a lack of data visibility and data fragmentation is an irresponsible way for any enterprise to operate, especially with the ongoing large-scale working-from-home [WFH] being practised today due to the Covid-19 pandemic.” he says.
Compounding these challenges are cyber security threats, says Justin Loh, Singapore country director for Veritas Technologies.
Increased reliance on cloud platforms and the ongoing WFH regimes have broadened cyber attack surfaces, particularly ransomware, he says.
By not having a centralised consistent view of data and easy accessibility to data by any and all of their applications that require it, they will never be able to maximise the value of their data
Ravi Rajendran, Cohesity
According to Loh, a Veritas ransomware resiliency report noted that 42% of organisations globally suffered some form of ransomware attack in 2020, raising concern for data privacy and security.
“Hackers do not discriminate as they just want your data and will do anything to get it in today’s climate, where the workforce and business environment have become more distributed,” he says.
Loh also says many ASEAN governments are rewriting legislation in their respective countries to include heavy fines for enterprises that are found guilty of data breaches.
“For these reasons, enterprises should think of how to leverage data protection and management tools that deliver security and visibility,” he adds.
The challenges facing many enterprises involve a lack of the right people to manage data and advanced tools for analytics, says Lian Jye Su, principal analyst for ABI Research.
“The rapid advancement of data acquisition, storage architecture and analytics tools means enterprises are playing catch-up,” says Su. “Many lack predictive analytics, dashboards and visualisation tools and skilled people to run them, and this is a serious challenge.”
From legacy to multicloud
According to Su, enterprises have to move away from legacy data management systems for three reasons. First, it costs more to maintain systems that are outdated and no longer being supported by suppliers. Second, the lack of support for older systems means data is prone to being hacked, stolen and manipulated. Third, data trapped in these systems are siloed, which prevents advanced analytics being used to make sense of it.
Cohesity’s Rajendran agrees, but notes that the transition between legacy and cloud is not a zero-sum game.
“To holistically manage data management issues, enterprises need to consolidate their data by leaving existing legacy infrastructure while embracing either a cloud data platform approach or an on-premise strategy with offload to the cloud,” he says.
Still, the move to the cloud is an extremely complex task and experts that Computer Weekly spoke to warn that cloud computing is not a silver bullet that will magically solve these challenges.
Rajendran says the key to building a successful cloud-based data management architecture lies in a good distributed file system, a single local control plane for easy management, and to deliver services through a collection of applications.
He says organisations should consolidate all their data management services and ensure that data and apps are migrated onto a software-defined platform to eliminate siloed infrastructure and visibility.
Then, IT teams will need to simplify operations by creating a single global dashboard, which allows policies to be set and action taken.
“Cloud providers can take care of physical cloud infrastructure such as datacentres and servers,” he says. “But the data is solely yours to secure, manage and back up.”
Veritas’ Loh says many enterprises use different point solutions to manage their legacy data and are often misled by the false notion that cloud data is backed up by their respective cloud providers.
“The propensity for security blind spots will inevitably increase,” he says. “Organisations need to address the data complexities with the mix of legacy infrastructure and cloud, firstly by getting visibility into the data that exists in their organisation.”
For a start, says Loh, enterprises can get their data off tape and onto the cloud. “Datacentre backup is the lowest hanging fruit, as it can capture the value it brings back to the business quickly,” he says. “Organisations can then further optimise data management and protection by looking into leveraging the cloud for disaster recovery.
“Other processes to look at after that are dynamic workload migration, backup and data protection in the cloud and maintaining data visibility across all environments, no matter where the data resides.”
Ecosytm’s Bird stresses that enterprises must ensure data protection by building systems with privacy by design. “Customers must be more aware and ask their cloud providers for greater transparency about how their data is stored and used,” he says.
ABI Research’s Su adds: “Make sure data security measures are sound, with clear efforts to maintain data integrity and prevent corruption or destruction of data.
“Also set up authoritative and clear guidelines for data management, focusing on policy enforcement, overall responsibility and governance authority.”
Where does AI fit in?
One area gaining interest in many enterprises that are already on cloud-based data management platforms is the use of artificial intelligence (AI) and machine learning to give them a further edge in today’s world.
Bird says machine learning can help alleviate the difficulty of integrating a diversity of data sources at speed. “And AI data mapping can be used to detect personally identifiable information and confirm it is complying with privacy laws, for example.”
Su believes many suppliers have started to offer natural language capabilities to perform advanced search and query by fetching data from cloud databases for training and testing purposes, and enterprises should consider these offerings.
Cohesity’s Rajendran says yet another use case is machine learning-based threat detection, which can reduce downtime with early discovery and detection of an intruder in an enterprise's production environment.
“Machine learning capabilities can be built into a data management platform, so that when the primary data change rate is out of the norm, indicating potential data encryption activity, the platform detects the irregular behaviour,” he says.
Meanwhile Veritas’ Loh argues that because so much of today’s data is located at the edge of the network, data protection stands to benefit from AI and machine learning, especially when it comes to predictive insights.
Loh says most IT managers are taking a “rear-view mirror” approach when reacting to unplanned downtime caused by interruptions related to software, hardware errors and component failures.
Incorporating predictive technologies will enable proactive monitoring for downtime and faults, so IT managers can take preventative action before a disruption occurs, he says.
Datacentre backup is the lowest hanging fruit, as it can capture the value it brings back to the business quickly
Justin Loh, Veritas Technologies
“Mundane tasks such as the classification of data for the right policies can now be done automatically,” says Loh. “With proper implementation, machine learning tools can help enterprises perform important information management tasks pertaining to self-configuration and self-tuning.”
But while the use of AI and machine learning in data management systems may be beneficial, it can also be a double-edged sword, warns Cohesity’s Rajendran.
Models that underpin machine learning initiatives could be imprecise, even catastrophic, without an adequate training mechanism, he says. “Without sufficient relevant data, a detrimental ripple effect may occur, with predictions pointing in an inaccurate direction.”
Loh says employees must be well equipped to use data analytics to derive meaning and build on insights. With data literacy skills, staff are more agile and responsive, honing their decision-making skills with actionable insights, he adds.
Best practices
As for how best to achieve success in data management, Loh stresses that the right people, processes and technology must be in place.
He says there should not be just one person responsible for managing data, but the task is cross-functional in nature.
“It starts with a change in mindset, where enterprises need to build a culture of data governance and compliance, starting with adopting good data habits at the employee level,” says Loh.
“Next up is to build processes and conduct training to empower employees with full visibility and control of data, as proper management of data is no longer the work of just the CIO or CDO, but a combination of all departments.”
Other success factors are: support from senior management, including a centralised strategy to manage data better; implementing appropriate and up-to-date technologies; and investing in data management initiatives.
Cohesity’s Rajendran recommends that enterprises break down the traditional walls that exist within an organisation so that CIOs or chief information security officers can consolidate information, which can then be used by top decision-makers to make informed and intelligent choices.
Doing so is also likely to see storage costs drop because removing deduplication frees up server space or cloud spend and enhances the overall security of any data management platform, while limiting the attack surface of corporate data, he says.
“Keep three key steps in mind – detect, prevent, and recover,” says Rajendran. “While productivity and business continuity are at the top of the list of priorities as businesses try to recuperate from the economic impact of the Covid-19 crisis, it is also vital to keep company data safe and secure while WFH arrangements are in place.”
Ecosytm’s Bird adds: “Given how fast the world has become data-driven, enterprises have become too dependent on technology vendors for data management. They will now need to build competencies internally so that they will not be restricted by their own capabilities.”
