Fotolia
DWP data hub develops prototype skills recommendation engine
Machine learning algorithm on Microsoft Azure identifies associated skills, based on the skills specified in online job adverts
The Department for Work and Pensions’ (DWP) data science hub has created a prototype skills recommendation engine, inspired by e-commerce sites such as Amazon. The service is designed to help jobseekers find similar roles based on their skills, experience and salary.
Helping people find jobs has always been one of the DWP’s key roles. “As a service, it has existed since 1909 through the Labour Exchange Act,” says Sean Massey, product owner for the Find a Job service at the DWP’s digital group. “When I started work, there were jobs cards at the jobcentre. In many ways, the current self-service site is the online version of that jobs board. But we are now looking at how to use data to improve outcomes, both for jobseekers and employers.”
Massey says the DWP’s job site service, which went live in May, replaced Universal Jobmatch. The service has so far seen 17 million job searches and 155,000 vacancies, ranging from someone who has a disability who needs personal care right through to the NHS and the likes of Goldman Sachs.
“We do not scrape other sites,” he says. “Like a lot of job sites, vacancies are posted by employers and we have a team to verify employers.”
In March, the Office for National Statistics warned that about 1.5 million jobs in England are at high risk of some of their tasks and duties being automated in the future. This has a direct impact on the recruitment market, which is beginning to recognise that people no longer have a job for life.
For Massey, the recruitment world is changing rapidly. “People are working much longer, the retirement age is going up and there is a reduced number of young people in the workforce,” he says.
Massey says that with initiatives such as #futureofwork, the recruitment industry is looking at a more skills-based approach to vacancies. “There is no longer a linear route to finding a job,” he observes.
The DWP’s jobs recommendation engine prototype project effectively widens the groupings of particular skills, providing opportunities both for jobseekers and employers who may be struggling to find skills in specific sectors.
Newcastle: home of the DWP data hub
Ryan Dunn is head of the DWP’s data science hub in Newcastle, and says it was set up to investigate how to support decision-making throughout government departments. “The primary way we work is user research and data discovery, leading to prototype products that solve problems,” he says.
The data hub has been working on a specific piece of work for the Department for Digital, Culture, Media and Sport (DCMS) on its Digital Skills Partnership (DSP) policy to increase digital skills in local areas.
Dunn says the initial idea for the skills recommendation engine was developed at the DataJam North East hackathon, which took place at the National Innovation Centre for Data last September. More than 200 individuals, from across the public and private sectors and academia, took part in the hackathon. One of the hack teams focused on skills using data from job search engine specialist Adzuna, which worked in partnership with the DWP on the Find a Job service.
Daniel Routledge, a data scientist at the DWP’s data hub, worked on the skills recommendation engine prototype and says the task in hand was to develop a proxy for job skills transferability.
The team needed to find a machine learning algorithm that could identify which skills were associated with particular job adverts and where different skills coexisted. For instance, says Routledge, “JavaScript is related to CSS and tends to go alongside it in job adverts”.
Read more about innovation at the DWP
- Juan Villamil, director of enterprise infrastructure and production operations at DWP Digital, discusses how the Department for Work and Pensions tests out new tech ideas.
- Department for Work and Pensions wants to reduce reliance on existing identity assurance methods such as Gov.UK Verify and face-to-face.
The DWP data hub team working on the prototype needed to apply a suitable artificial intelligence algorithm in order to understand the relationship between different skills. “This is quite computationally intensive,” says Routledge. “It could take 1,500 hours and 275GB of RAM. We tried using a tiny sample of 80,000 job adverts and even that took many hours to run.”
The team selected the FP (Frequency Pattern) Growth algorithm, which Routledge says is easier to scale across distributed nodes, compared with other algorithms.
“FP Growth effectively spits out a series of common combinations of skills, then extracts association rules, which basically says that if you see these certain skills, then suggest other [associated] skills,” he says.
The Microsoft Azure public cloud platform was selected to run the algorithm using Azure Databricks and PySpark (Python Spark) across a distributed computing platform. According to Routledge, by using a large number of computing nodes, it was possible to process 50 million job adverts in 63 minutes. “And it only cost £4 of resources because we could spin up nodes quickly and spin them down again,” he says.
The project is currently in the alpha phase of its development and Routledge anticipates that the skills recommendation engine will ultimately be embedded into the Find a Job a website, providing a pick list, where the jobseeker selects a skill and gets job titles that match. There is also the possibility of creating an application programming interface (API)-based service, which could be embedded into other websites, to provide skills recommendations.
