The path to true data democratisation

This is a guest blogpost by Ed Thompson, CTO and co-founder, Matillion.

Jessica is a data analyst at a leading eCommerce company and is tasked with analysing the usage patterns of the business to work out which ones are worth focusing on, based on consumer behaviour. As a person with an analytical mind, she enjoys the challenge of finding insights and patterns within data. It’s not the task of inferring experiences with data that’s giving her the Monday morning blues, it’s what she has to do to achieve it.

Jessica understands the data inside her organisation with all of its history and nuance. She can see how valuable it would be and she understands what she needs to do to get that value from it, and where to find it. The problem is the data needs plenty of blending and preparation and in order to do that, she has to go to the data team in the IT department because they have the access, and the tools, to do what Jessica needs. However, the IT department does not have the knowledge or expertise of the data sources that Jessica is managing. So now Jessica has days, if not weeks of chasing, cajoling and explaining to do before she gets results, and it won’t be quick!

Like Jessica’s employer, companies use data to make better business decisions and gain a competitive edge. The cloud and cloud data warehouses (CDWs), have brought data and analytics to a new threshold for modern businesses. CDWs let enterprises integrate, transform, and analyse data at speeds previously unheard of. But now, you can do it at tremendous scale and use all of your valuable data, not just as much as limited technology and resources can handle.

In the current business era, enterprises are experiencing diverse data, and a changing technology landscape. That’s not to say that the need for data is going away – IT data professionals still need enterprise-scale and powerful transformation capability, and always will. However, at places like Jessica’s company, where data is a competitive advantage, there is a new persona within the business that doesn’t necessarily work in IT. This persona, known as the Citizen Data Professional, can be found in departments like marketing, sales, finance, and other disciplines across the business. They want to use data to solve challenges in their areas of expertise.

The market talks a lot about democratizing data, or the ability to make data available to internal users, both technical and non-technical. While “data democratisation” is a trend, the concept is invaluable for organizations of all sizes. Equipping the whole business, from your engineers to business analysts, with data insights enable an organization to achieve the level of data-centric thinking required to sustain competitive advantages. This allows a business to be truly data driven.

However, there has been a technological limitation in achieving this goal. Current options in the market assume all data is owned by IT and all end users have a proficient understanding of ETL (extract-transform-load) technology. Although from small start-ups to large global enterprises, this is not always the case.

To truly democratise data, an enterprise must have technology solutions in place that enable it to consolidate data. These are capabilities that new software solutions are only just now beginning to offer. The strain on businesses is compounding as the technical skills gap grows. Gartner predicts that, by 2020, 75% of organisations will experience visible business disruptions due to infrastructure and operations skills gaps.

Enterprises can resolve these challenges by adopting solutions that democratise data access, allowing more individuals like Jessica to break down data silos. With a simplified UI and code-free pipeline builders, anyone with access to the data source and an authenticated CDW can manage their own data analytics needs. This capability both empowers business users and frees up developers to focus on high-value activities to progress the organization’s data maturity from data preparation to transformation to machine learning and automated insights.