UK needs coordinated effort to regain ground in high-performance computing

A review conducted by the Government Office for Science looking at large-scale computing has warned that the UK is losing ground in the league of supercomputing and faces challenges in acquiring software development and high-performance computing operations skills.

The authors of the Large-scale computing: the case for greater UK coordination report pointed out that the explosion of new data was driving new application areas, which potentially have significant implications for the large-scale computing ecosystem, changing the balance of computing infrastructure required as demand for powerful computers increases.

However, they warned that “many potential new users may not have the prerequisite awareness or skills to seek out or make use of large-scale computing resources”.

In the preface to the report, the government’s chief scientific adviser, Patrick Vallance, discussed the role the government needed to play in nurturing and supporting the UK ecosystem, both as a consumer and a funder of large-scale computing systems. “Our primary recommendation is to establish a team within the government to take policy responsibility for large-scale computing,” he said.

The report noted that the UK’s large-scale computing infrastructure lags behind other major global economies. As of November 2020, China and the US had 214 and 113 of the world’s top 500 computer systems respectively, and France and Germany had 18 and 17. In contrast, the UK had 12. The UK’s share of global high-performance computing (HPC) capacity has decreased by three-fifths over five years, falling to 2% in 2019.

Future computing capabilities offer the opportunity to help to solve problems that are currently intractable. Projects are underway around the world to develop exascale computing, which would deliver systems roughly 140 times faster than the UK’s fastest system as of November 2020.

“These systems could mark a step-change in certain capabilities. Meanwhile, novel computing architectures could greatly accelerate specific applications such as training AI algorithms,” the report said.

Looking at the challenges high-performance and large-scale computing face in the UK, the report warned that there was an acute shortage of large-scale computing professionals, including system architects, data engineers, system operations professionals and software engineers.

It noted that career paths for computing professionals were often not well-defined in government and academia. “Exploitation of advanced computing requires skilled cross-disciplinary teams. Career pathways should be developed to help attract, support and retain these skillsets,” the report stated.

“Job security, salary structures and progression opportunities should be improved to retain talent. The pipeline of talent could be bolstered through expanded apprenticeships and professional training programmes.”

In terms of programming skills, the report said significant training and support was required before inexperienced programmers could efficiently make use of parallel computing at scale. It recommended that domain scientists should be involved in mapping real-world scientific problems to computational solutions.

From an IT operations perspective, the report discussed the complexity associated with running large-scale computing systems. “Highly skilled systems administrators and software developers are required to configure, tune and deploy software for use on large-scale computing infrastructure,” it stated.

“Typically, very few jobs will use 100% of a system, so multiple jobs are often run simultaneously using differing proportions or partitions of a system. Efficient job allocation is therefore essential to achieve cost-effective utilisation of systems. This creates complex scheduling and dependency challenges.”

To broaden accessibility to HPC and large-scale computer systems, the report discussed how cloud computing represented one mechanism available to small and medium-sized enterprises (SMEs) and those organisations where the costs of such powerful systems cannot be justified. For example, it said the Fortissimo project has created a successful marketplace for European SMEs to access large-scale computing resources, expertise, software and tools on a pay-as-you-go basis.