ONS publishes technical learnings of UK’s first online census

The Office for National Statistics (ONS) has outlined the technical learnings of the UK’s first census delivered fully online.

According to the ONS document, uptake of the digital census was high, with 97% of households in England, Wales and Northern Ireland taking part. Some 22 million census responses were received between February and June 2021 through an online electronic questionnaire (eQ).

Some 19 million user sessions were recorded by the Census 2021 questionnaire. Since the counts were recorded through Google Analytics cookies, the ONS estimated that the user sessions and page views were significantly higher, at potentially around 3.3 billion.

The high demand experienced by the ONS for the census – with nearly nine in 10 households completing it online in a short period of time and just under half a million submissions received per hour at the peak of the period – were met through the use of cloud architecture.

Some 468 instances of the eQ application were in use to manage the high levels of demand, each with one Docker container. The ONS team chose not to use auto scaling due to the potential creation of latency and the impact on users. In total, 885 containers were provisioned with a further 225 reserved in case of need.

According to the ONS document on the digital census experience, the eQ product served 5.8 billion HTTP requests during the five-month period. A total of 1,097 HTTP errors arose. This is the equivalent to a 0.00001862% error rate, and there were no HTTP errors on Census Day.

Live service risks had to be managed during the expected Census Day peak. According to the document, a flatter usage curve would be preferred to stabilise system demand and avoid peaks. On the other hand, the Census Day peak was predictable and the ONS team was able to prepare for it with the flexibility of the cloud-based architecture and infrastructure, as well as the ability to scale that setup if required.

The ONS pointed out that the success of the digital census demonstrates that “large government digital services can be securely delivered in-house using cloud architecture and agile development”.

On agile delivery, the document said this type of in-house development method was well-suited to deliver the census, mainly due to the need to understand the implications of operating a live, scalable service at the same time as it was being built. Running a live service and iterating it over the years would not have been possible for the 2021 census, since the technologies underpinning it were only required for a specific period and consistency in delivery was crucial.

Additionally, the document noted the digital census was designed against potential cyber security attacks. The platforms were secured against data exfiltration attacks, which would have caused the potential loss of citizens’ online response data, as well as service disruption attacks and events, such as distributed denial of service attacks and website defacing.

Live operations at the ONS also included a security operations centre supporting phishing site detection, which would initiate the takedown of sites attempting to scam the public into thinking they were being followed up for non-completion of the census.

The digital-first census also informed the ONS about the online behaviours of people using the service, the document pointed out, and this could be helpful in future large government digital service delivery.

Average completion time for the questionnaire was 23 minutes, but large proportions completed it in 16 to 17 minutes. Most people (56.4%) completed the survey through mobile devices, while 34.6% used desktops and 8.9% used tablet devices.

According to the ONS team, the high online completion rate exceeded expectations, and also demonstrates positive advances in digital inclusion. “From this we are better able to understand the willingness of citizens to interact with national government services online and how they do so,” the document noted.