DWP completes in-house VME replacement project
The department has taken an in-house approach to swapping its virtual machine environment (VME), getting rid of legacy infrastructure and replacing 11 key benefits systems
The Department for Work and Pensions (DWP) has built its own, in-house virtual machine environment (VME), moving away from a long-standing outsourcing deal.
The huge project, believed to be the largest of its kind in Europe, has seen the department develop and roll out an in-house VME replacement, replacing 11 key benefit systems which pay out more than £150bn a year to citizens, without any downtime.
This has all taken place in the middle of the pandemic, which hasn’t been an easy feat for the team at DWP.
Computer Weekly spoke to DWP Digital programme lead Mark Bell and his technical lead Nick Cutting about the project, why they decided to move it in-house and the challenges of completing a huge transformational project in the midst of a pandemic.
The DWP’s old VME mainframe services were fully managed by Fujitsu since the system was first installed in 1974, and most of the department’s critical IT systems, including the benefit systems, were still being run on the proprietary operating system, originally developed by ICL before its acquisition by Fujitsu.
Bell described the old ICL infrastructure as “the proverbial burning bridge” that needed to be replaced.
He said that the department takes a digital-by-default approach, but that “when you’re always tied to back-end systems that are so hard to integrate with, it’s hard to do”, so bringing it in-house made complete sense and aligned with the department’s overall digital strategy.
The DWP first began plans for a VME replacement in 2014, and in August 2015 it decided to issue a prior information notice, looking to procure a full range of VME services, taking a tower-based model approach to the replacement. However, in 2016 the programme was reborn as an in-house project. With help from its support partner Advanced, DWP took the reins completely.
Cutting said that the decision to bring the VME in-house was “definitely a brave decision”, but also the right one. “What it did bring was flexibility,” he said, adding that it has led to the department being able to do things it “never could have done, or that would have taken significant time at a significant cost” if the VME had still been outsourced.
Lockdown challenges
The project to replace the VME has been ongoing for a few years, but just as the Covid-19 pandemic hit, the team was in the thick of it.
During the period between April 2020 to January 2021, the team delivered four of the largest systems on the platform.
Bell said that the pandemic was definitely not something that was in the risk management plan, and the first lockdown came just as the team was about to deliver its biggest service migration.
Suddenly, the team had to get to grips with remote working, embracing collaboration tools and holding their morning scrum meetings virtually.
DWP’s biggest benefit service, which is the jobseeker’s allowance service, was due to be migrated shortly after lockdown began, and the service was successfully migrated over Easter 2020.
“We had to migrate somewhere in the region of 6.3 billion rows of data in less than 24 hours,” Bell said.
“The planning and the activities that we had around that was based on us actually being physically in the office to go through those implementation plans and make sure they worked. All of a sudden, we find ourselves working remotely and our ways of working had to dramatically change,” he said.
“As Covid came along, the pressure on not just us getting the implementation right, because the disaster if we got it wrong – and the number of new claims, both jobseekers allowance claims, and Universal Credit claims that were coming in as a result of Covid – meant there was an increased dependency on us getting this right.
“There was a lot of consideration about whether we were doing this at the right time. Should we be taking this level of risk? Because this is national critical infrastructure for the country, and if this fails what would happen?”
However, there was also a huge risk in doing nothing. Analysis done by Cutting on the old VME infrastructure showed that it was already in a “dangerous position” and was running at a high level of CPU utilisation, so bringing on an extra workload of 1.5 million claims could have been disastrous.
“If we didn’t migrate, not only were we in danger of not being able to deal with the new claims, but actually some of the old claims we potentially couldn’t support. It came at the right time for us. We were ready as a programme to take this service to step it up, which gives us increased bandwidth, increased storage capability and increased processing capability,” Bell said.
Cutting added that from a programme perspective, by that point they were “a well-oiled machine, and if this had happened at the very beginning of the programme, it would’ve been a different story”.
The project to migrate jobseekers allowance ended up being so successful that the team won a Civil Service Award for it.
Life’s ambition
The project has seen around 200 staff work to complete it, both contractors and DWP’s in-house team. Bell said that the team working on the VME replacement has been fantastic and very experienced.
He added that for some of the staff “this was probably their last hurrah before they actually retire” and that some “have come out of retirement to do this as well”.
“There’s a real conviction and commitment within the programme. For some, this was a career ambition. For those people who were brought up on these services and have designed, architected and coded them, to now do something after 30-40 years – where you’re not just replacing them, but putting them into a new infrastructure whereby we can exploit the data that’s been locked away for years, we can integrate with frontline services – that’s great.”
Bringing this in-house is also future-proofing it. Cutting said the department has now moved from both “bordering-on-being-obsolete technology and going down a cul-de-sac in terms of where to go”.
“We have, over the years, struggled to unpick that technology and in particular services. We’ve gone to a virtualised platform now, so potentially this could go on forever again. We’ve significantly improved the documentation and we’ve built a new integration capability so we can open up, enhance it and build on it to our heart’s content,” Cutting said.
“We view this as being a building block to support the department’s modernisation agenda going forward.”
Project complete
On 22 March, the programme board delivered its last senior delivery report and the programme was closed down, having completed the goals set in the outline business case.
“Not only have we done this, we have actually rebuilt a lot of capability. We’ve invested heavily in training for individuals,” Bell said.
Although it’s been a huge project, end users may be none the wiser, because the front-end user experience hasn’t changed. The batch processing, however, has.
“Although these are still batch services, we are seeing some of our batch services not running four or five hours, but running in under an hour. So we’re seeing incredible performance improvement as well,” said Bell.
He is hugely proud of his team. “We replatformed 11 major benefit payment services all part of the national critical infrastructure with the blink of an eyelid as far as our users are concerned, as far as our operators are concerned and indeed as far as citizens are concerned. This was almost the best kept secret in digital government,” he said.
DWP’s contract with Fujitsu officially came to an end on 23 February 2021 and its VME is now officially department-run. Overall, the cost of the programme was £143m by the end of March 2021, with projected savings by 2028/29 of £257m against licencing and hosting costs. This means that overall, affordability and net present value of the programme stands at £149.8m.
Its choice to bring its VME in-house is likely to be seen as brave by many in government. Another government department, HM Revenue and Customs (HMRC), recently signed a further five-year deal with Fujitsu for its VME platform.
Fujitsu has been running HMRC’s VME for more than 30 years, and the department felt there was “no alternative that would technically be capable of being successfully deployed” and that “only Fujitsu could provide the services”.
Read more about government and IT
- Scottish government announces artificial intelligence strategy that involves the creation of a Scottish AI Alliance to be drawn from across society, including trade unions and companies.
- Home Office has completed its deployment of Oracle’s Cloud Applications suite to all its functions across 35,000 staff, ahead of other government departments.
- The Home Office has failed to acknowledge and be transparent about Digital Services at the Border programme problems, according to the Public Accounts Committee.
