MEPs vote in raft of amendments to EU AI Act

Initial reactions

Responding to the amendments, the Computer & Communications Industry Association (CCIA Europe) – whose members include the likes of Meta, Google, Amazon, BT, Uber, Red Hat and Intel, among many other tech firms – said that although there were some “useful improvements”, such as the definition of AI being aligned to that of the Organisation for Economic Co-operation and Development (OECD), “other changes introduced by Parliament mark a clear departure from the AI Act’s actual objective, which is promoting the uptake of AI in Europe.”

It specifically claimed that “useful AI applications would now face stringent requirements, or might even be banned” due to the “broad extension” of prohibited and high-risk use cases: “By abandoning the risk-based structure of the act, Members of the European Parliament dropped the ambition to support AI innovation.”

CCIA Europe’s policy manager, Boniface de Champris, added that the association is now calling on “EU lawmakers to maintain the AI Act’s risk-based approach in order to ensure that AI innovation can flourish in the European Union.

“The best way for the EU to inspire other jurisdictions is by ensuring that new regulation will enable, rather than inhibit, the development of useful AI practices.”

Tim Wright, a tech and AI regulatory partner at London law firm Fladgate, similarly noted that the AIA would “may take the edge off” European AI companies abilities to innovate.

“US -based AI developers will likely steal a march on their European competitors given news that the EU parliamentary committees have green-lit its ground-breaking AI Act, where AI systems will need to be categorised according to their potential for harm from the outset,” he said.

“The US tech approach (think Uber) is typically to experiment first and – once market and product fit is established – to retrofit to other markets and their regulatory framework. This approach fosters innovation whereas EU based AI developers will need to take note of the new rules and develop systems and processes which may take the edge off their ability to innovate.

“The UK is adopting a similar approach to the US, although the proximity of the EU market means that UK-based developers are more likely to fall into step with the EU ruleset from the outset; however the potential to experiment in a safe space – a regulatory sandbox – may prove very attractive.”

Civil society groups that have been campaigning around the AIA, on the other hand, welcomed a number of the new amendments, but warned there are still a number of issues, particularly around industry self-assessment and carve-outs for national security or law enforcement.

Griff Ferris, senior legal and policy officer at non-governmental organisation Fair Trials – which has been explicitly calling for a ban on the use of AI and other automated system to “predict” criminal behaviour since September 2021 – described the prohibition of predictive policing as a “landmark result” that will protect people from an “incredibly harmful, unjust and discriminatory” practice.

“We’ve seen how the use of these systems repeatedly criminalises people, even whole communities, labelling them as criminals based on their backgrounds. These systems automate injustice, exacerbating and reinforcing racism and discrimination in policing and the criminal justice system, and feeding systemic inequality in society,” he said.

“The EU Parliament has taken an important step in voting for a ban on these systems, and we urge them to finish the job at the final vote in June.”

Ella Jakubowska, senior policy adviser at European Digital Rights (EDRi), added: “We are delighted to see Members of the European Parliament stepping up to prohibit so many of the practices that amount to biometric mass surveillance. With this vote, the EU shows it is willing to put people over profits, freedom over control, and dignity over dystopia.”

Leufer similarly welcomed the two committee’s amendments, which he said better protects peoples rights: “Important changes have been made to stop harmful applications like dangerous biometric surveillance and predictive policing, as well as increasing accountability and transparency requirements for deployers of high-risk AI systems.

“However, lawmakers must address the critical gaps that remain, such as a dangerous loophole in Article 6’s high-risk classification process.”

Self-assessment

Speaking with Computer Weekly ahead of the vote, Leufer said Article 6 was previously amended by the European Council to exempt systems from the high-risk list (contained in Annex Three of the AIA) that would be “purely accessory”, which would essentially allow AI providers to opt-out of the regulation based on a self-assessment of whether their applications are high-risk or not.

“I don’t know who is selling an AI system that does one of the things in Annex three, but that is purely accessory to decision-making or outcomes,” he said. “The big danger is that if you leave it to a provider to decide whether or not their system is ‘purely accessory’, they’re hugely incentivised to say that it is and to just opt out of following the regulation.”

Leufer said the Parliament text voted on by the two committee’s includes “something much worse…which is to allow providers to do a self-assessment to see if they actually pose a significant risk”.

EDRi shared similar concerns around Article 6, noting it would incentivise under-classification and provide a basis for companies to argue that they should not be subject to the AIA’s requirements for high-risk systems.

“Unfortunately, the Parliament is proposing some very worrying changes relating to what counts as ‘high-risk’ AI,” said Sarah Chander, a senior policy adviser at EDRi. “With the changes in the text, developers will be able to decide if their system is ‘significant’ enough to be considered high risk, a major red flag for the enforcement of this legislation.”

On high-risk classifications generally, Conor Dunlop, the European public policy lead at the at the Ada Lovelace Institute, told Computer Weekly that the requirements placed on high-risks systems – including the need for quality data sets, technical documentation, transparency, human oversight, et cetera – should already be industry standard practices.

“There’s been a lot of pushback from industry to say that this is overly burdensome,” he said, adding that a solution would be to simply open more systems up to third-party assessments and conformity checks: “I think that would compel safer development and deployment.”

State overreach

Regarding the prohibitions on live and retrospective facial recognition, Leufer added while the Parliament has deleted all the exemptions on the former, it has not done so for the latter, which can still be used by law enforcement with judicial authorisation.

“Any exception means that the infrastructure needs to be there for use in those exceptional circumstances. Either that requires permanent infrastructure being installed in a public space, or it requires the purchase of mobile infrastructure,” he said. “They’re not going to leave it sitting around for three years and not use it, it’s going to be incentivised to show results that it was a worthwhile investment, and it will lead to overuse.”

Pointing to a joint opinion on the AIA published by two pan-European data protection authorities, Leufer added that those bodies called for a ban on remote biometric identification in any context, and clearly stated that both live and retrospective facial recognition are incompatible with Europe’s data protection laws.  

“It’s already illegal, we [at Access Now] been saying that for a long time, so it would be good if the AI Act put it to rest and had an explicit prohibition,” he said. “Anything less than a full ban is actually worse than not having anything, because it could be seen as providing a legal basis for something that’s already illegal.”

Leufer added part of the problem is that lawmakers have fallen into the trap of seeing live facial recognition as somehow more dangerous than retrospective facial recognition: “There is something visceral about being matched on the spot by this thing and then having the instant intervention, but I really think the retrospective is much more dangerous, as it weaponises historic CCTV footage, photos, all of this content that’s lying around, to just destroy anonymity.”

There are also concerns about the AIA allowing the development and deployment of AI for national security or military purposes with no exemptions on its use.

In a conversation with Computer Weekly about the ethical justifications of military AI, Elke Schwarz – an associate professor of political theory at Queen Mary University London and author of Death machines: The ethics of violent technologies – for example, described the AIA’s approach to military AI as “a bit of a muddle”.

This is because while military AI systems are exempt from the requirements if specifically designed for military purposes, the vast majority of AI systems are developed in the private sector for other uses and then transferred into the military domain afterwards.

“Palantir works with the NHS and works with the military, you know, so they have two or three core products of AI systems that obviously change based on different data and contexts, but ultimately it’s a similar logic that applies,” she said.

“Most big ambitious AI regulations end up weirdly bracketing the military aspect. I think there’s also a big lobby not to regulate, or let the private sector regulate ultimately, which is not very effective usually.”

In a legal opinion prepared for the European Center for Not-for-Profit Law in late 2022, emeritus professor of international law at the London Metropolitan University Douwe Korff said: “The attempts to exclude from the new protections, in sweeping terms, anything to do with AI in national security, defence and transnational law enforcement contexts, including research into as well as the ‘design, development and application of’ artificial intelligence systems used for those purposes, also by private companies, are pernicious: if successful, they would make the entire military-industrial-political complex a largely digital rights-free zone.”

Describing the national security exemption as “a huge potential loophole”, Ferris also noted it would “undermine all other protections” in the AIA, “particularly in the context of migration, policing, and criminal justice, because those are all issues which governments see as issues of national security”.

Access Now and EDRi are also calling for the national security and military exemptions to be dropped from the AIA.