Data-driven decision making will fail – and here is why

Drowning in data

In 2010, we created 2ZB (zettabytes) of data globally. This year, more than 100ZB will be created. That’s about 50 times more – and roughly equivalent to every human generating an entire copy of the Library of Congress each year.

If data-driven decision making was right, this growth should lead to vastly improved organisational performance.

Ideally, you might hope for a 50x increase in performance since 2010. Has that happened? Clearly not. In fact, has there been any improvement since 2010? If there has, it is not obvious.

In hundreds of conversations with businesses, no one has ever said to me: “Data has made my life easier. Suddenly, I have all the answers.” In reality, conversations are more like: “Data is overwhelming. It’s all in silos. People cherry-pick bits to help their cause.”

So what is going wrong?

The history of decision-making

Let’s go back in time to understand why. For thousands of years, it was thought that understanding, and therefore decisions, came from divine mandate. About 400 years ago, philosophers realised that collecting data to create understanding was a good thing.

However, they also thought data alone was sufficient to establish how the world was and predict what would happen next – a process called induction. They thought a wider understanding of what was going on didn’t matter.

Notice this is the same claim made for data-driven decision making – but we know a wider understanding does matter.

Will stars appear in the sky because they did yesterday? Well, yes – for a while. But at some point, they will burn out. What was an obvious extrapolation is, suddenly, no longer true.

This view changed with the philosopher Karl Popper, who said we don’t extrapolate inductively from data, because that’s impossible. In fact, we guess what’s going on, then find data to falsify that theory.

This is a crucial change. Suddenly, the focus is the theory – not the data. This means the theory can be very different from an extrapolation from data.

So, the first stage of decision making was divine truth is everything, the second stage was data is everything, and the third stage was theories are proposed and refuted through data.

Businesses have gone from stage one to stage two over the past few decades. Shifting from CEO mandates to looking at data is simple enough – but we still haven’t made the leap from data to theories.

Why this matters

For the first time ever, machine learning offers us the chance to help people build more complex explanatory theories, test these to see what is likely to happen, and see what we can do about it. That sounds abstract without an example, so let’s take the pandemic.

Understanding when Covid cases will rise and fall has driven decisions affecting trillions of pounds and billions of lives.

In March 2020, the NHS needed to make life-or-death decisions on the back of understanding whether curves were trending up or down. For the first time ever, they connected data flows from across the entire system.

That is a potentially overwhelming amount of data – but, crucially, they weren’t paralysed by this sudden torrent. They didn’t succumb to data-led decision making. In a pandemic, that would have been catastrophic. Why was that? 

Well, working alongside the NHS, we applied approaches from a nascent field called decision intelligence. We used this data to build the understanding the NHS needed to make confident decisions about which hospital wards to open or close, and which resources to send where. 

Rather than just showing data on current levels of patient demand, we could also show demand for tomorrow or next month, the reasons why demand was changing, how that affected resources such as beds or ventilators, and what actions they could take.

Rather than making decisions on data alone, the NHS made them on an understanding of how and why events were unfolding. The collective results have been widely credited with saving thousands of lives.

Decision intelligence is the future

Decision intelligence is one of the most important contemporary applications of artificial intelligence (AI). This next generation of technology will have far-reaching consequences for how organisations make decisions, and it comes at a much-needed time.

We need to move beyond the spreadsheet hell of data-led decisions. We need a revolution in how our leaders approach decision making.

And, just as Popper taught us, we need to move to decisions based on understanding – and that means true decision intelligence.

Marc Warner is CEO at Faculty, which he co-founded in the belief that the benefits of AI should extend to everyone. He has since overseen the growth of Faculty to one of Europe’s leading AI companies. He has led many data science projects, with clients ranging from multinational companies such as easyJet and Siemens to the UK government and the NHS. Outside of Faculty, Warner advises the government via the AI Council.