Big data skills shortages – and how to work around them

Skills shortages are the perennial headache of CIOs everywhere, particularly when looking to develop leading-edge big data, analytics and artificial intelligence (AI) systems.

For the fourth year running, the Harvey Nash/KPMG CIO Survey has found that big data and analytics are top of the skills shortage critical list. This is having a significant impact on all organisations, with two-thirds of IT leaders saying it is preventing them from keeping up with the pace of change.

Given the newness of the discipline, it is not surprising that there aren’t enough skilled data scientists. Very few universities offer pure data science degrees (as opposed to computer science). Many schools still don’t even offer computer science at A-level or GCSE. It’s going to take some years before there are enough skilled data scientists in the workforce. 

Skills shortages present a real problem and we need to accelerate mechanisms to address them. However, there are ways in which IT leaders can mitigate the issue. 

One of the key questions CIOs often ask themselves is: do we buy or build? In fact, with data analytics, you can rent. Because data analytics, big data and AI are new areas, there is a natural tendency to think that the organisation has to prove itself and build its own system.

But companies such as Amazon Web Services (AWS), Google, Microsoft, IBM and others are building applications for voice recognition, image classification, facial recognition and more that are available for rent in the cloud with no long-term commitment.

One example is Google’s AutoML. Google’s systems automatically create a machine learning model from content uploaded by the client. Rather than in-house teams spending a lot of time fine-tuning algorithms in search of the best solution, this – and similar services being developed by other organisations – could save time and ease the pressure on resources.

I believe CIOs need to find people with a slightly different skillset – what you might call “bridgers” or “shapers”. These people are the ones who can engage with the business, understand what it needs and where the problems and friction points are – and then start shaping use cases to solve it.

In other words, they are not merely technically skilled data scientists, but the missing link or bridge between the IT team and the business – the communicators and facilitators who can understand and articulate the real need, and so drive greater efficiency. This is a gap that exists in nearly every organisation I go to.

CIOs should not forget that, these days, there are whole data science courses openly available online. The man who many regard as the godfather of deep learning, Geoffrey Hinton at the University of Toronto, has a course available. There is one from Stanford University, too.

Everyone is wrestling with a shortage of talent. Everyone fears the business next door has just hired that all-important guru. This won’t change any time soon, but through a variety of approaches CIOs can at least dial down the size of the problem.