Public Health England builds Ceph and Lustre cloud for analytics

Public Health England (PHE) has put open source storage at the centre of its IT strategy, with a private cloud built on Red Hat’s Ceph object storage distribution and the DDN-supported Lustre parallel file system for high-performance computing (HPC) analytics.

The move has allowed it to avoid supplier lock-in and build a petabyte-scale storage environment across three sites in southern England to support PHE’s healthcare analytics.

PHE was created in 2013 from more than 70 agencies to provide the NHS and other public and private organisations with scientific support in the sphere of public health.

It employs around 5,500 people and core to its activities are data analysis across three key areas.

The first is bioinformatics, which analyses DNA for surveillance of infectious diseases and requires high-throughput computing with many small jobs and lots of CPU and disk input/output (I/O).

The second is modelling and economics, which runs real-time simulations to predict expected disease dynamics.

Third is emergency response, which runs multiple models to predict risks posed by infectious disease threats.

The latter two need mainly traditional HPC, but with lots of CPU and low to moderate disk I/O.

When PHE was formed, it needed to build a shared IT infrastructure for the newly created body.

Windows and VMware were in use for business-as-usual workloads, but choices had to be made over what file system environment would be used for its bioinformatics HPC environment, said Francesco Giannoccaro, head of high-performance computing and infrastructure for PHE.

“[Dell EMC] Isilon was high performance, but quite expensive and not built for high rates of parallel I/O, more for virtual machines. [IBM’s] GPFS was in use with about 40% of the world’s top 500 supercomputer users, but the skillsets weren’t necessarily there,” said Giannoccaro.

PHE went with the Lustre file system on DDN SFA10K hardware for the bioinformatics HPC cluster. Giannoccaro was later asked to begin work on an HPC environment for the whole organisation.

A key consideration was that the three main HPC clusters would not run at full utilisation 24/7/365. For this reason, among others, the organisation settled on a Red Hat virtualisation environment with the KVM hypervisor and an HPC cloud based on OpenStack. This cloud architecture allowed each HPC cluster to burst to additional processing capacity when needed.

In this cloud environment, the PHE scientific data catalogue was developed using iRODS, an open source technology that helps organise, catalogue, aggregate and share large sets of data.

Storage-wise, the bioinformatics 1,500-core Lustre-based cluster is on DDN ES7K hardware with 400TB of storage capacity (plus 500TB of archives).

Meanwhile, the emergency response and statistics and modelling clusters – with 3,000 cores – are on Red Hat Ceph in the HPC cloud that runs on Lenovo blade servers with 500TB of storage and 8PB of archiving capacity.

Flash is used for metadata for Lustre while the Ceph environment comprises hybrid flash tiers of storage.

“Lustre is the technology that’s most widely used for I/O-bounded large amounts of data,” said Giannoccaro.

He said some of the advantages of going down the open source route are that it “reduces vendor lock-in and is open in a way that other organisations can access and share information”

For its cloud infrastructure, Ceph is the storage behind the OpenStack environment.

“It’s the most widely diffused for OpenStack,” said Giannoccaro. “We wanted to find the right balance between the right technology and something that would be supported in a robust way by the right partner.”

The Ceph platform is a software-only product based on multiple storage nodes and a technology called Rados (reliable autonomic distributed object store) that lays out and manages data across multiple clusters.

Red Hat packages and sells a commercial version of Ceph, marketed as Red Hat Ceph Storage.

Ceph supports S3, Swift and native object protocols, as well as providing file and block storage offerings.

Lustre is an open source parallel distributed file system built for for large-scale Cluster computing. Lustre’s open licensing and high performance mean it is commonly used in supercomputer applications.

Giannoccaro said PHE’s next moves would be towards use of containers to deliver and rapidly scale content to its web front end using Red Hat’s OpenShift container management platform.