AI can never be given control over combat decisions, Lords told

Automation bias

Then there is the added problem of automation bias (referring to the tendency for humans to trust the outputs of automated systems more than they would trust information from another source), the complete elimination of which Nolan said would be a “pipe dream”.

“It’s an extremely active and long-running area of human factors research on how to reduce automation bias or eliminate it, and we don’t know how,” she said.

On whether an AI-powered weapon would ever be able to autonomously assess proportionality of combat decisions, Nolan said she believes it is “absolutely impossible” for a machine to make those kinds of determinations, as only a human could assess the overall strategic context.

“You need to know the anticipated strategic military value of the action, and there’s no way that a weapon can know that,” she said. “A weapon is in the field, looking at perhaps some images, some sort of machine learning and perception stuff. It doesn’t know anything. It’s just doing some calculations which don’t really offer any relation to the military value.”

Explaining how AI models mathematically allocate pixels to identify the contents of images – which any AWS would have to do in the field from a live feed – Taniel Yusef, a visiting researcher at Cambridge University’s Centre for the Study of Existential Risk, said that although the underlying maths could be “accurate”, that does not necessarily mean the outcomes will be “correct”.

Giving the example of when she tested an algorithm designed to distinguish between images of cats and dogs, Yusef said that even simple tasks such as this can and do go wrong.

“It decided the cat was a dog,” she said. “What concerns me is, when this happens in the field, you will have people on the ground saying these civilians were killed, and you’ll have a report by the weapon that feeds back, ‘But look at the maths’.

“The maths says it was a target that was a military base … because the maths says so, and we defer to maths a lot because maths is very specific, and the maths will be right. There’s a difference between correct and accurate. There’s a difference between precise and accurate. The maths will be right because it was coded right, but it won’t be right on the ground,” said Yusef.

“So when you ask the question about proportionality and if it’s technically possible [to delegate responsibility to AI], no, it’s not technically possible, because you can’t know the outcome of a system, how it will achieve the goal that you’ve coded, until it’s done it, and you don’t know how it’s got there,” she said.

Peer interjection

When a peer interjected to say humans could make similar mistakes, as “the other day, I saw a dog which I thought was a cat”, Yusef replied: “You didn’t shoot it.”

Christian Enemark, professor of international relations at the University of Southampton, said: “The autonomous discharging of [discrimination and proportionality] to a non-human entity is a philosophical nonsense, arguably.”

He added that it should always be a human agent that makes decisions and takes responsibility for them, and that the general conversation about AWS should be expanded out to include other practical areas where they could be used.

“Weapons can be used outside of armed conflict, and yet the conversation has been primarily directed towards armed conflict and the law that governs armed conflict, which is international humanitarian law,” he said. “But it need not be so restricted, and arguably it ought to be expanded to include the use of violence by the state, for example, in law enforcement purposes – we need to be thinking about what the implications of AI incorporation might be in that context.

“And once we get out of the context of armed conflict, we’re not restricted to talk about humanitarian law. We’re open now to be inspired and guided by international human rights law as well.”

In its first evidence session, the committee heard that the potential benefits of using AI in weapons systems and military operations should not be conflated with better international humanitarian law compliance, on the basis that speeding up warfare beyond the ordinary cognition capabilities of humans would limit people’s ability to prevent an unlawful or unnecessary attack.

The expert witnesses in that session also noted that the deployment of AI weapons could also make the use of violence more rather than less frequent, because the threshold of resorting to force would be significantly lower.