Why an ecosystem helps IT projects move forward

One of the key messages to come out of the recent Cloudera event in London is the idea of working with a data ecosystem of IT providers to achieve a business objective.

In a presentation discussing the role of the IT ecosystem, Merv Adrian, principal of IT market strategies, said organisations should start using ecosystems. “Boards are prepared to take risks,” he said. “They believe transformation is necessary and possible. But only 20% say they are getting anywhere with transformation. The way we make it work is with market-leading partners working together, to collectively transform the systems we have today in the most cost-effective way.”

Adrian said 69% of organisations believe they have too many analytics platforms. Data, he said, is also siloed and organisations run multiple relational databases. This leads to complexity; there is no single version of the truth and it becomes difficult to derive value from data across multiple data sources.

According to Adrian, using open standards encourages data sharing. “When we liberate resources, we are able to do new things,” he added.

In Gartner’s The impacts of data ecosystems report, published in April, the analyst stated that data management is no longer wholly focused on relational data. “Document, graph, time-series, wide-column, key-value, ledger and other targeted data stores all provide specific optimisations for different types of data and different use cases,” the authors of the report wrote.

A data ecosystem is seen by many as providing an approach to pre-integrating data from different sources to streamline delivery and optimisation to support business decision-making.

This is how Legal and General Investment Management (LGIM), which also appeared onstage at Evolve 2023, has used Cloudera. Wulstan Reeve, head of data marketplace at LGIM, said the data team wanted to provide relevant data to the business and clients faster, but that pulling in the right data for people to access is a complex task. Providing relevant data implies frictionless access. The data, he said, must also be accessible in the format people require. Then, there’s the challenge people face in finding out what datasets are available, which requires data discovery. Data integrity and accountability are the other requirements LGIM needed to factor in.

To support the data strategy set by the company’s chief data officer, the team needed to specify the capabilities required from a data platform with the company’s tech strategy, which is about being cloud-first. Stuart Toll, senior enterprise architect at LGIM, said that time to market, integration time and skills were among the criteria used to assess the data platform providers. For Toll, while LGIM could have probably made any data platform work, he said “we are an asset management firm”. “We buy where we can and only build to differentiate.”

This influenced the company’s data integration strategy. LGIM did not want to be in the business of stitching lots of tools together, as Matt Bannock, head of data engineering at LGIM, explained. “I don’t think there are bad tools on the market anymore,” he said. “There are so many good players, but what is key is time to market.”

Bannock said that with some tools, IT departments need to spend time on data integration. “Being able to just start working with the data, start running the calculation and start generating the output is much more valuable to us than the potential half a percent advantage we could achieve if we created our own ecosystem,” he said. “There’s a lot of benefit in buying into an ecosystem.”

While the company does adopt best-of-breed tools where appropriate, it has bought into the Cloudera ecosystem, which offers a data platform that supports hybrid environments as well as multi-cloud unified security, governance and metadata management.

Gartner notes that data today is distributed, disparate and diverse. The experiences of data and analytics experts that Gartner spoke to suggests there is significant integration effort needed to ensure the IT system where data resides is able to work seamlessly.

According to Gartner, the industry is starting to respond, with cloud service and software providers developing more mature data ecosystems as the market moves from “some assembly required” to a more integrated platform experience.

As LGIM has found, the level of integration now available means IT teams can avoid doing much of the hard work stitching these disparate data sources together to provide a data marketplace for the business.