UK government coronavirus data flawed and misleading

More ways of measuring Covid cases

David Paton, professor of industrial economics at Nottingham University Business School, has regularly published figures based on the actual date of death in the English health service. He reckons the figures are “reasonably complete” only five or six days afterwards. By 6 May, NHS England had reported 281 deaths taking place on 30 April, 90% of Paton’s current total of 313 – which includes one report made as late as 20 September. The government now publishes its own UK-wide “date of death” data, reporting 548 deaths on 30 April.

Looking at when deaths actually occurred provides a different view of the pandemic’s spring peak. According to both the original and revised version of the government’s data, coronavirus death reports peaked on 21 April, a Tuesday. Both Paton’s and the government’s numbers based on date of death peaked almost a fortnight earlier, on Wednesday 8 April. However, the “date of report” daily figure remains the one most used, probably because it is immediately available and doesn’t change as more reports are made.

There are other credible ways to measure coronavirus deaths. By 18 September, the government’s all-UK death toll had reached 41,801. But the Office for National Statistics (ONS) reckons approximately 57,677 people had died by that date across the UK where death certificates mentioned Covid-19 – including 769 on 30 April – and that 53,663 more deaths had occurred in England and Wales alone in 2020 compared with the average of the previous five years, a measure known as excess deaths.

The government’s other main measure – positive tests for cases of coronavirus – has bigger problems. On 21 September, the official number of new daily cases exceeded April’s record and has continued to climb. But while the rise is a cause for concern, it should be seen in the context of daily test numbers having quadrupled since April. “The supply side has changed,” says Paton.

Case data on positive tests is a side-product of the testing system. It took time for the government to set up large-scale public testing; testing centres are likely to be established in areas where cases are growing; and in recent weeks demand has outstripped supply. The ONS is carrying out weekly research by sending tests to a random sample of the population, but this is smaller-scale, slower and lacks local detail.

Furthermore, cases differ in significance: 200 cases in households across a city are more threatening and harder to control than the same number in one university hall of residence. This does not mean positive test case data is useless, just that it needs treating with caution. Paton says it has the advantages of being easy to understand, quick to react and is highly localised. “But when you’re making very serious and significant policy interventions that affect everybody’s everyday life, my view is it’s not quite good enough,” he says.

More data means greater accuracy

Paton advocates using a holistic range of data, including the different measures of deaths and case numbers, NHS data on hospital admissions and triage assessments, and data from a symptom-reporting app run by health science company Zoe. All have their flaws, and the more reliable ones – hospitalisations and deaths – are slower to react. They work best together.

“When all those things are moving in the same direction, that probably tells you an increase in cases is genuine, not just due to testing,” he says. “If you rely on any one of those, you may make false policy decisions,” he says.

Nigel Marriott, an independent statistician who provides consultancy and training, says using a range of sources will be vital in tackling increasingly localised outbreaks, with urban areas outside southern England worst affected: “The effort needs to be focused on those areas,” he says. But most of the data published by the government is not localised, so Marriott looks for rising case numbers in neighbouring areas as corroborating evidence.

“There’s a classic trade-off between quality and speed. Often, the best data that helps you make a decision will not be available for a week or so”

Marriott believes the pandemic has demonstrated a basic tension in gathering and using data. “There’s a classic trade-off between quality and speed,” he says. “Often, the best data that helps you make a decision will not be available for a week or so.” Using fast data can be justified, but he thinks the government should explain the limitations of such measures.

Openness is vital, Marriott adds, noting that in June the government imposed extra local regulations in Leicester, partly on the basis of unpublished “pillar 2” coronavirus test results carried out by commercial labs for the general population. Public Health England – which the government announced in August would be replaced by a new National Institute for Health Protection – has not consistently published its raw data, something Marriott describes as “a giveaway” that an organisation is not confident of data quality.

“It’s a very 2010 thing to say, but [we need] more open data,” says Gavin Freeguard, programme director at the Institute for Government think tank, calling for a return to a policy of last decade’s coalition government. He says that some of the best uses of coronavirus data have been made by third parties, such as Matthew Somerville’s local lockdown lookup service.

Freeguard adds that the government’s recent publication of a national data strategy and work to recruit a chief data officer are positive signs that people at the centre of government recognise the value of good data. But he warns there is a “danger of technology and data solutionism”. The backlash caused by qualification bodies using algorithms to decide grades for pupils after examinations were cancelled may have damaged public trust in data use, he thinks.

Using data visualisation for impact

Freeguard, who writes an email newsletter on data visualisation, says the government has made effective use of graphs to communicate data on coronavirus, but that there has been “a lack of uncertainty”.

People may prefer a simpler message – “I think there is something about we the public trying to reach for certainty in uncertain times” – but this can be misleading. It can be done: the ONS, which Freeguard praises for its work on Covid-19, illustrates the uncertainty with its estimate that 103,600 people in England had the virus in the week from 13-19 September by also including 85,600 and 123,400, the range it is 95% confident the true figure is in.

Andy Cotgreave, technical evangelism director at visualisation software provider Tableau, says graphs showing how cases would grow and peak given different transmission rates helped make the case for the UK government to enact strict controls on people. He describes “flattening the curve” as “possibly one of the most influential visualisations ever”, by communicating how a virus can spread with different rates of transmission.

But Cotgreave criticises a graph used by the UK’s chief scientific adviser, Patrick Vallance, and chief medical officer for England, Chris Whitty, at a press conference on 21 September. This projected cases doubling every seven days from 3,105 on 15 September to nearly 50,000 by 13 October. Vallance and Whitty stressed it was a scenario, not a prediction, but the single set of bars on the graph suggested otherwise.

“Was it the best case, the worst case, the mid-case?” says Cotgreave. This is part of a bigger problem, he adds: “The risk is that people are fatigued with data, trust is waning and these things could backfire.”

Nottingham University’s Paton argues that the scenario of doubling every seven days was far more than Spain and France – which the two scientists had just discussed – were experiencing, with those two countries seeing cases double approximately every 20 days.

“I called it sleight of hand and I don’t think that’s unfair,” he says. The graph seemed to be a way to generate headlines that would justify new regulations announced by the Prime Minister the next day. “That’s not what I think independent scientists should be doing.”

Jennifer Rogers, the Royal Statistical Society’s vice-president for external affairs and head of statistical research and consultancy for UK-based biometrics researcher Phastar, says the government needs to maintain trust in how it deals with data on coronavirus. She says that while it makes sense to change the way measures work when limitations become apparent, this should not be undertaken lightly and needs to be explained openly and honestly.

Rogers believes the government should consider presenting less data. During the period of national lockdown, the same graphs were used every day, often with minimal changes: “As a statistician who likes data, I felt it was too much data,” she says.

Passing on numbers and showing graphs may seem to boost the credibility of someone presenting, but it can also overwhelm those watching. “There’s a big lesson: just present what’s important, what’s relevant and what people really need to know in order to make decisions about their day-to-day lives,” she says.