Public sector buyers of AI tech must interrogate its suitability

Taskforce launch

Although official use of foundation model applications like ChatGPT in the public sector is currently limited to demos, prototypes and proofs of concept (noting there is some evidence that individual civil servants are using it on an informal basis), the Institute noted the UK government is actively seeking to accelerate the UK’s AI capabilities through, for example, the launch of its £100m Foundation Model Taskforce in June 2023.

Given this push, much of the briefing document focuses on the leading role the private sector has played in developing the technology, and how public sector bodies can navigate their own procurement and deployments in a way that limits the various harms associated with it.

For example, the Institute said there is a risk that being overly reliant on private technology providers can create a misalignment between applications developed for a wide range of clients and the needs of the public sector, which generally handles much larger volumes of sensitive information.

There are also further risks arising from automation bias, it said, where users are overly trusting of the foundation model outputs, or treat them as if they are from a human.

It added that the wide range of risks currently associated with foundation models are best dealt with by the upstream providers of foundation models at the training stage through, for example, dataset cleaning, instruction fine-tuning, or reinforcement learning from feedback.

As such, the Institute said all public bodies seeking to procure external foundation model capabilities should require detailed information about the associated risks and mitigations upfront when procuring and implementing.

“For example, when procuring or developing a summarisation tool, public-sector users should ask how issues like gender or racial bias in text outputs are being addressed through training data selection and model fine-tuning,” it said. “Or when deploying a chatbot for public inquiries, they should ensure the process of using data to prompt the underlying large language model does not violate privacy rights, such as by sharing data with a private provider.”

To support more effective foundation model governance, the Institute added policymakers should regularly review and update the relevant guidance; setting procurement requirements to uphold standards, which should be written in tenders and contracts; requiring data used by the models to be held locally; mandate independent third-party audits of the systems; and pilot limited use cases before a wider rollout to identify all risks and challenges.

It added that effective governance of AI needs to be underpinned by the Nolan Principles of Public Life, which includes accountability and openness.

However, in July 2023, the Institute warned that the UK government’s “deregulatory” data reform proposals will undermine the safe development and deployment of AI by making “an already-poor landscape of redress and accountability” even worse.

In a report analysing those proposals – largely contained in the March 2023 AI whitepaper – it found that, because “large swathes” of the UK economy are either unregulated or only partially regulated, it is not clear who would be responsible for scrutinising AI deployments in a range of different contexts.

This includes recruitment and employment practices, which are not comprehensively monitored; education and policing, which are monitored and enforced by an uneven network of regulators; and activities carried out by central government departments that are not directly regulated.

“In these contexts, there will be no existing, domain-specific regulator with clear overall oversight to ensure that the new AI principles are embedded in the practice of organisations deploying or using AI systems,” it said, adding that independent legal analysis conducted for the Institute by data rights agency AWO found that, in these contexts, the protections currently offered by cross-cutting legislation such as the UK GDPR and the Equality Act often fail to protect people from harm or give them an effective route to redress.

“This enforcement gap frequently leaves individuals dependent on court action to enforce their rights, which is costly and time-consuming, and often not an option for the most vulnerable.”

Risks and opportunities

In September 2023, the House of Lords launched an inquiry into the risks and opportunities presented by LLMs, and how the UK government should respond to the technology’s proliferation.

In his written evidence to the inquiry, Dan McQuillan, a lecturer in creative and social computing, highlighted the risks of outsourcing of decision-making processes to such AI tools.

“The greatest risk posed by large language models is seeing them as a way to solve underlying structural problems in the economy and in key functions of the state, such as welfare, education and healthcare,” he wrote.

“The misrepresentation of these technologies means it’s tempting for businesses to believe they can recover short-term profitability by substituting workers with large language models, and for institutions to adopt them as a way to save public services from ongoing austerity and rising demand.

“There is little doubt that these efforts will fail,” said McQuillan. “The open question is how much of our existing systems will have been displaced by large language models by the time this becomes clear, and what the longer-term consequences of that will be.”