There are many reasons why data streaming, or real-time data analytics, is becoming important to the future of business, but one of the most obvious ones is speed.
This would be music to the ears of someone like serial entrepreneur and author Gary Vaynerchuk, who talks a lot about the importance of speed in business, claiming it’s “four billion times more important than perfection”.
As Gartner analyst Rita Sallam similarly said during a presentation at the Gartner IT Symposium/Xpo earlier this year, “in the face of unprecedented market shifts, data and analytics leaders require an ever-increasing velocity and scale of analysis, in terms of processing and access, to accelerate innovation and forge new paths to a post-Covid-19 world”.
But what does this really mean? As a business, you always want to know more about your customers and to find ways to be more efficient. In the right hands, real-time analytics offers you that chance – and there seems to be no limit to where it can make a telling impact.
Perhaps it’s no surprise, then, that by 2022 most business systems will, according to Gartner, feature real-time data capabilities. Business intelligence is demanding it, and as we look to 2021 and the prospect of a post-pandemic world, we will see an appetite for technologies that can combine to enable truly personalised, live, informed decisions to be made.
Data analytics in action
In some ways, we are already seeing elements of this, not least at one of Europe’s most advanced airports, Rome’s Fiumicino, or Leonardo da Vinci, Airport. A Tibco customer, the airport’s owner Aeroporti di Roma has been steadily overhauling its digital infrastructure to enable the flow and management of data from all aspects of the site – reservations, parking, check-in, security, shopping, boarding, public transport, car rental and so on.
“We are focusing on passenger flow analysis to understand customer needs, and how to improve the time they spend in various areas,” says Floriana Chiarello, head of demand management at Aeroporti di Roma. “We used Tibco to develop the passenger flow service. Heat maps and bubble maps show passenger volumes and density, as well as typical passenger paths.
“The gates are the last points at which we capture Wi-Fi data, so we can correlate passenger flow and flight data with boarding card information used in our shops. By analysing correlated data, we can activate predictive analysis and study trends and historical buying patterns to identify future improvements for the airport’s shopping and food businesses.”
It’s an example of how real-time data flows can revolutionise the way organisations learn from and interact with customers. When we emerge from Covid-19, the devil in this sort of detail will be essential for businesses to navigate some turbulent waters ahead.
As an SAS report, Experience 2030: The future of customer experience is now finds, customers, more than ever before, are going to be a little picky when it comes to parting with their money. A third of UK customers would ditch companies after just one poor experience, says the report, while 90% would abandon companies after just two to five poor examples of customer service, before moving to competing brands.
That may seem a little harsh but that’s the reality. But it’s not just about customer experience. Real-time data analytics is already having an impact on decision-making across industries, cutting down decision-making time as well as making those decisions more accurate and lasting.
A good example of this is in Zurich, one of the world’s smartest cities. Thanks to live streaming data analytics through the Vianova mobility data platform, the city has managed to increase transport policy response times. The result has been the creation of two new dedicated cycle lanes and a dedicated parking infrastructure for e-mobility, improving multi-modal journeys, the quality of transport services and reducing waiting times.
“In transport, changes in people flows caused by post-pandemic trends, including working from home and avoiding modes of mass transit, and higher parcel delivery volumes are highlighting the need for real-time analytics at the local authority level,” says Vianova co-founder and CEO Thibault Castagne.
“If we want smart cities to be anything more than rhetoric, they will need to be built on the foundations of data sharing and real-time analytics”
Thibault Castagne, Vianova
“This is particularly vital for accommodating the growth of micro-mobility alternatives such as e-scooter and bike-sharing services. In many cases, multiple private operators are involved, so live data sharing helps them to work with local authorities to ensure the safe, effective deployment of their services. If we want smart cities to be anything more than rhetoric, they will need to be built on the foundations of data sharing and real-time analytics.”
What is clear is that real-time analytics is going to touch almost every industry at some point over the next few years. According to Russell Gammon, chief innovation officer at Tax Systems, this includes the UK government’s drive towards making the tax system digital by 2022.
“Tax data analytics provide insight into existing and live tax data using natural language processing (NLP) which interrogates the data at source, such as directly from ERP [enterprise resource planning] or accounting software,” says Gammon.
“This provides real advantages. For instance, with regards to compliance, it allows you to evidence claims assessments to provide proof of entitlement so that you can quickly respond to any challenges from HMRC [HM Revenue and Customs]. You can also ensure you are maximising tax relief, tracking appropriate tax rates across different entities, and keeping on top of deadlines.”
As well as financial services, there are countless examples of pioneering data streaming solutions from internet of things (IoT) devices in energy through to cyber security, manufacturing and, of course, healthcare.
As Jon Payne, sales engineering manager at InterSystems UK, puts it: “Smart data means smart decisions. Many businesses have understood that they need to capture data, increasingly in real time, to better understand how their business is operating and to make it easier to adapt.
“However, understanding better what is the right data to retain and leverage is a challenge. Capturing and retaining ‘everything’ and putting it into cheap object stores just creates swamps that become increasingly difficult to draw value from, let alone leverage in an adaptive manner. So being smart about what to collect and how it’s maintained and used is crucial.”
Challenges – if it’s so good, why isn’t everyone streaming?
The problem facing many organisations is how to create the necessary infrastructure and culture that can deliver the data required.
As a survey of IT decision-makers by data integration firm SnapLogic recently found, some businesses are struggling. There is, according to the research, a distrust of data that is undermining progress, with 84% of IT decision-makers agreeing analytics projects are delayed due to data not being available in the required format. With 77% of organisations not fully trusting their data, it’s worrying that 54% admit to relying on this poor-quality data to drive strategic decision-making.
According to Simon Cole, CEO of Automated Intelligence, this is not entirely surprising. “Legacy software, combined with vast quantities of unstructured data, represents the greatest challenge to organisations using data for real-time decision-making,” he says. “Most companies are not even aware of the terabytes (and often petabytes) of unstructured data floating around in their systems.”
“Data analytics needs to be a part of the operational strategy, so you find just the right historical data you need to help make better decisions. It is analytics-driven data management and automation that makes this possible”
Krishna Subramanian, Komprise
This is worrying if leaders, perhaps under pressure to deliver data-driven trends and analysis, are forging ahead using tools and platforms not entirely built for the job. It’s a common problem because not everyone has the budget to rip and replace systems. It’s almost inevitable that, at some point, legacy IT and data siloes will get in the way of progress.
Krishna Subramanian, chief operating officer at Komprise, says most data visualisation tools today focus on structured data databases, and that “since 90% of the world’s data is now unstructured, we need more advances in analysing and visualising unstructured data”.
That makes sense. As Gartner predicts in its Top 10 trends for data and analytics in 2020 paper, the traditional data dashboard will decline in use, as “dynamic data stories with more automated and consumerised experiences will replace visual, point-and-click authoring and exploration”.
The shift to in-context data stories means the most relevant insights will stream to each user based on their context, role or use, Gartner adds, leveraging technologies such as augmented analytics, NLP, streaming anomaly detection and collaboration.
Subramanian agrees with the sentiment. She adds that businesses often look at data analytics as a reporting and business intelligence activity for management that is siloed and separate from the day-to-day operations. “This results in analysis that does not drive actionable results,” she says. “Data analytics needs to be a part of the operational strategy, so you find just the right historical data you need to help make better decisions. It is analytics-driven data management and automation that makes this possible.”
Certainly, artificial intelligence (AI) techniques such as machine learning (ML), optimisation and NLP are needed to manage data infrastructures and deliver insights and even predictions. As Gartner predicts, investment in automated data platforms will only increase, and this will lead to further change in how data is streamed and managed in the future. The challenge now is to make better use of the data companies do have and try to improve speeds to enable better streaming.
As InterSystems’ Payne recently found during some company research, “only 11% of retail, consumer packaged goods and manufacturing organisations have access to data that is less than an hour old, meaning there is a significant delay in their ability to use this data to make timely and informed decisions”.
Clearly, there are huge benefits to streaming live data, but for the moment, at least, this remains a fragmented and complex industry, but one that is on the leading edge of change. Any business that gets it right now will be able to face the post-pandemic future with a certain degree of well-informed optimism.
