Protecting children by scanning encrypted messages is ‘magical thinking’, says Cambridge professor

Language modelling is error-prone

Levy and Robinson proposed “language models running entirely locally” on a mobile phone or a computer to detect “language associated with grooming” in their paper, an idea that has now appeared in draft UK and European Union (EU) laws.

But the use of natural language processing (NLP) models to detect illegal speech – whether sexual grooming, terrorist recruitment or hate speech – is highly error-prone, Anderson argued. Analysing text content alone, it is difficult to achieve error rates significantly below 5-10%. The European Commission assumed that it might get 10% false positives in an internal document presented to the European Council in June. This would mean a billion false alarms would have to be manually processed each day.

There is only one trial scanning tool available, created through Microsoft’s Project Artemis which is supplied through Thorn, a private US company. There is no independent assessment of the technology’s effectiveness and Microsoft’s claims about it are “notably weak”. Anderson argued that it was difficult to see how anyone could trust a natural language text scanning tool that was trained on data to which the public has no access.

There are too many ways that machine learning can be subverted – for example, by manipulating the inference engine, the training data or its labelling. It is entirely unclear, he said, how mandatory remote scanning of messages on user devices could ever be “squared with the human right to privacy and the established ban on mass surveillance without warrant or suspicion”.

Manual moderation of false positives not feasible

Tech companies need to deal properly with complaints from users but frequently fail to do so because employing human moderators is expensive, said Anderson. 

They are willing to act on takedown requests from police and from copyright lawyers because ignoring them can be expensive. But they do ignore requests from ordinary users, including women and children, who may wish to report abuse. “That needs to be fixed, whether by criminal sanctions or by significant financial penalties,” the paper stated.

If natural language processing technology suggests there is a 60% probability of a child being groomed, this passes the civil threshold for liability if a technology platform fails to act. If costs were not an issue, a human moderator would be able to call the potential victim, and if the victim agreed that grooming had taken place, an investigator could work with them to preserve forensic evidence.

But human intervention costs money, and given the current state of the law and industry practice, it is more likely that the suspect, and potentially their family and friends, would have their accounts blocked.

Levy and Robinson’s claim that false positives from machine learning algorithms can be dealt with by human moderation is “simply wrong”, Anderson argued. This is because law enforcement agencies and technology companies have little or no liability for the human cost of false positives.

That was the case when a father took a photograph of his young son, at the request of a nurse. The image was uploaded automatically by his phone to Google. He was visited by the police and, despite being cleared, has been unable to gain access to his Google accounts.

If large numbers of accounts are referred for moderation, either because of false positives or children sending “sexting” messages to other children, then a significant number of young people may lose access to their digital lives.

Evidence for benefits of electronic surveillance is weak 

Government ministers in the UK have criticised Facebook’s plans to move Facebook Messenger to end-to-end encryption.

Anderson said it was possible that arrest numbers might fall if Facebook Messenger introduced end-to-end encryption, but the evidence to support that is weak, as the number of people arrested or interviewed for image offences has remained constant despite the proliferation of end-to-end encryption services.

Evidence from Germany suggests that electronic surveillance has not helped law enforcement in practice. The country collected communications data between 2008 and 2010, stopping when the practice was found to be unlawful. But statistics show that police access to the data in Germany did not have a deterrent effect, did not assist in clearing up crimes and did not increase convictions.

“When governments argue that privacy must be impaired once more in the hope of arresting more abusers, some scepticism is called for,” Anderson argues.

Better methods of crime prevention

Levy and Robinson’s focus is on child abuse, but Britain’s Online Safety Bill would also empower authorities to impose server-side scanning of text for terrorist radicalisation and recruitment.

The history of function creep in surveillance suggests that client-side scanning for terrorist content would be the next logical step, if it is permitted for child protection. However, research shows there are strong links between violent political extremism and violence or misogyny against women.

“Where ministers buy the idea of a magical software ‘solution’, as the industry likes to call its products, the outcomes are often disappointing and sometimes disastrous”

Rather than using bulk interceptions to look for young men who “express an opinion or belief supportive” of a prescribed group or download a copy of The Anarchist Cookbook, the police need to pay more attention to violence against women and children, says Anderson.

“If there are 200 youths hanging out in a mosque with a fiery Salafist preacher, the man to watch is not the one who downloaded the [Anarchist] cookbook, but the one who beat up his sister,” the paper argued.

Anderson claimed client-side scanning was not the best solution if the policy aim is to prevent abuse to children.

Violent crime against children is associated with deprivation, unstable families living in slums, structural unemployment, despair leading to alcohol and drug abuse, and gangs providing role models, he said.

Better ways to tackle the problem would be to increase child benefit and increase the number of child social workers.

Police need to be able to focus on the primary problem, which is preventing and prosecuting crimes against children and political violence.

Good policy requires government to coordinate multiple bodies in the state and private sectors by working to get the incentives right. For police and social workers, protection work is a hard and thankless task, and it is easy to be distracted by technical solutions, the paper argued.

“The idea that complex social problems are amenable to cheap technical solutions is the siren song of the software salesman and has lured many a gullible government department on to the rocks,” wrote Anderson.

“Where ministers buy the idea of a magical software ‘solution’, as the industry likes to call its products, the outcomes are often disappointing and sometimes disastrous,” he said.