Home Office details seven-point plan to cutting its Immigration Technology team's cloud costs by 40%

The Home Office could cut its cloud costs by at least 60% in due course by doubling-down on a programme of IT resource optimisation that has already generated 40% in savings over the past year.

The savings have been accrued by the department’s Immigration Technology team, which consists of 500 developers working on various immigration-focused software projects.

An evaluation of how the team worked revealed the developers were not using the cloud resources at their disposal in the most efficient way possible, because of how focused they were on hitting their mission-critical project deadlines.

“By focusing on mission-critical deadlines, developers were using more expensive on-demand services and not planning resource usage,” said the department, in a blog post where it shared details of its cloud cost-cutting strategy.

“This led to expensive cloud bills, sometimes called ‘bill shock’, because most cloud providers charge for cloud resources by the second.”

To address this, the team got to work on researching industries practices that could help bring down its cloud costs, leading to the publication of a seven-point plan detailed in the blog post that – to date – has generated savings in the region of 40%.

One of the biggest sources of overall cloud cost savings came through ramping up the department’s use of excess cloud capacity in the form of spot instances, whereby providers enable end users to access spare compute capacity during off-peak times at discounted rates.

It is claimed this step alone resulted in the department paying around 80% less for its cloud resources.

The downside of this setup, however, is that the provider can request the resources back at any time, if another customer wants them and is willing to pay more, which is why the department opted to use these instances for non-production workloads and temporary jobs.

“It’s perfect for non-production services or temporary jobs, which currently make up about 30% of the compute powering their non-production containerised clusters,” the blog post said.

In a similar vein, the department also began to experiment with the use of reserved instances, whereby they receive access to discounted cloud resources by specifying how much they would need and paying upfront for them in advance.

Using this technique meant the department benefited from an upfront discount of 40%, but there were limitations on when, where and what conditions these could be achieved under.  

For example, the organisation also hit on the idea of scheduling services, such as compute and storage, to run only as and when needed to make the most of the per-second billing models its cloud suppliers offer.

The cloud systems underpinning the department’s development and testing environments were pinpointed as a prime candidate to test this out on, as they are never used overnight or at weekends. This, in turn, helped cut costs by a further 60%, it is claimed.

However, it is not possible to combine the use of reserved instances with scheduled services, because the cost-savings from the latter are accrued through switching the servers off for a period of time.

“Immigration Technology also found the savings from the up-front usage commitments were often lower than from many of the other techniques,” the blog post added.

“For example, turning off servers overnight and at weekends can save over 60% on cloud costs. Immigration Technology realised it could save more money by only using this method for services where no other techniques would work.”

The blog post goes on to detail several other steps the department employed to drive down its cloud costs even further, which included embracing auto-scaling, re-architecting some of its services to make them run in a cloud-native way, which included elements of containerisation, and making sure they kept their environments clean and tidy too.

The latter step generated a relatively small 1% cost saving, and involved encouraging developers to spring clean their testing environments by deleting or halting the cloud services they used to run them once they had completed this part of the development process.            

The plan is now to build on this work in the Immigration Technology team by spreading the word about how these cost savings were achieved through “show and tell” education sessions aimed at product managers and service teams.

It is also creating reports, the blog post added, and circulating these to other teams in the department that detail how its cloud usage-related efficiency gains have been achieved.

“By continuing these techniques, the team is confident it can increase cloud cost savings by at least another 20% a they continue to experiment,” the blog post said.