The data warehouse – why it’s time to stop writing its obituary and modernise your approach
This is a guest blogpost by Dave Wells, practice director, data management at Eckerson Group.
If there’s one thing the IT industry is exceptionally good at it, it’s proclaiming the death of a particular technology. In the mid 1980s industry observers sagely pronounced that COBOL was dead. Fast forward to today and COBOL still playing a role in healthcare for 60 million patients daily, 95% of ATM transactions, and more than 100 million lines of code at the IRS and Social Security Administration alone. I can’t help but recall Mark Twain’s famous quote, ‘the reports of my death have been greatly exaggerated!’
It’s not only COBOL that people want to consign to history. In 2013 SQL was declared dead, yet thousands of SQL job postings can be found on the web today. Just recently I heard that popular programming language Ruby was on its last legs. And then we have the data warehouse: over the last few years, there’s been a steady stream of obituaries announcing that the data warehouse was about to be consigned to the technology graveyard. But when surveys such as that conducted by Dimensional Research show that 99% of respondents see their data warehouse as important for business operations and 70% are increasing their investment in data warehousing, it appears the data warehouse remains very much alive.
But here’s the issue, while the data warehouse is alive, it also faces many challenges today. The root of the “data warehouse is dying” claim comes from the opinion that it hasn’t ever completely delivered on its promised value. The original vision was a seductive one – got a ton of data but no way to leverage it? No problem. Put it in a data warehouse and you’ll be extracting valuable insights to drive competitive edge in hours. Except, you couldn’t. Companies found that using traditional and very manual tools and processes, building and managing data warehouses wasn’t quite as easy as promised. Once built, typically, the data warehouses didn’t scale well, weren’t particularly agile or easy to rely on (due to performance variability), and, later on as needs evolved, they weren’t particularly well equipped for coping with the challenges of big data.
Data warehousing in the cloud
But, but, but…. The very fact that so many companies have clung doggedly to their (imperfect) data warehouse tells us that they are extracting some value. It’s just that it could be so much more. Enter the data warehouse of the cloud computing age. By migrating to the cloud, some classic data warehouse challenges disappear. Can’t scale or be agile in providing data quickly to those who need it? The cloud data warehouse changes that. Need to deploy rapidly but also dial up (and down) investment? The cloud data warehouse allows you to do that. And if you’re faced with the argument that the cloud erodes confidence in data governance and compromises the reliability of the data warehouse, well, there’s an answer to that too.
However, if we’re to constructively stem the expert proclamations of data warehouse demise, we must re-evaluate the original simplistic expectations of data warehousing as a one-size-fits-all, never evolving data infrastructure model for every organisation to reach its best use of data. Data warehousing must be fluid as organisational needs change and new data technologies and opportunities arise. And to accomplish that, we need to modernise how IT teams design, develop, deploy and operate data infrastructure. Expensive, redundant, laborious and time-intensive efforts intertwined with the use of traditional, non-automated approaches have limited organisational value greatly and cast a heavy cloud over data warehousing. However, organisations using automation software, such as Wherescape’s, to develop and operate data warehouses are providing far-reaching value to business leaders at greater speeds and less cost, while at the same time positioning IT to more easily incorporate timely technologies, new data sources and flex as business needs demand. With these adjustments, the reality of the data warehouse can better live up to the associated vision, and continue to deliver much more to organisations for many years to come.