UK police told to pause AI use in court cases over accuracy risks

The pause on generative AI

Police forces across England and Wales have been instructed to pause several uses of generative artificial intelligence in criminal justice work. This intervention specifically targets tasks such as preparing statements for court proceedings. The move comes as authorities grapple with the risks of using commercial AI systems in high-stakes environments.

The primary concern involves the lack of accuracy, provenance, and accountability inherent in current AI models. Experts argue that these systems are being deployed in situations where the origin of information is critical to the legal process. Because the stakes involve human liberty, the authorities are prioritizing reliability over the speed of technology.

Alex Murray, head of the Police.AI centre, stated that forces need to slow down to ensure these systems are properly assessed. The goal is to prevent AI from being integrated into the justice system before there are safeguards in place. This pause serves as a cooling-off period to establish how the technology can be safely managed.

The danger of plausible falsehoods

Generative AI presents a unique challenge because it can produce statements that sound highly credible while being entirely false. Unlike a human error, which might be caught by context, an AI can generate convincing misinformation based on patterns in its training data. This becomes dangerous when such statements are used to influence an active investigation or form part of a legal case.

A notable example occurred when West Midlands Police used Microsoft Copilot to prepare material regarding a proposed ban on football supporters. The AI referred to disorder at a previous match between the two clubs, but that match had never actually taken place. This incident serves as a warning that AI-generated content can look like legitimate intelligence even when it is completely fabricated.

The core issue is that large language models do not verify if a fact is true before they generate a response. They simply predict the next likely piece of information based on the data they were given. Consequently, the output can be authoritative in tone without being reliable in content, making it difficult for officers to distinguish between fact and fiction.

Evidence and the audit trail

In the criminal justice system, evidence must have a clear provenance, meaning it is possible to establish where it came from and how it was obtained. Generative AI complicates this because it does not provide a reliable audit trail for its reasoning. Even if a user asks an AI to explain its logic, the explanation itself may be a generated response rather than a true record of the process.

This lack of transparency creates significant hurdles for the legal system. Defense lawyers may need to question whether AI was used, what specific information was fed into the system, and whether every claim was checked against original evidence. If a record of the interaction with the system is missing, it becomes difficult to verify the integrity of the entire case.

The risks are already manifesting in real-world legal issues. A detective in Derbyshire is currently under criminal investigation following allegations that an AI chatbot was used to create evidential material. Several rape convictions are being reviewed because of allegations that the detective used the chatbot to produce paperwork intended to influence charging decisions.

The trade-off of human oversight

The most obvious safeguard against AI errors is human verification by a suitably qualified person. This person must check AI-generated material against original evidence and independently verifiable facts. However, this requirement creates a difficult trade-off between the time saved by using AI and the time required to double-check its output.

In the context of policing, this may be a necessary cost to ensure that responsibility remains with a human being. An AI-generated document can look official even if no one has verified its contents. If an officer simply decides that an AI output looks reasonable, they are not providing the same level of protection as a thorough manual check.

The focus must remain on ensuring that someone is held accountable for the accuracy of the information. Human oversight cannot simply mean having a person involved in the process at some point. It must mean that a human is responsible for establishing that the information is traceable and fit for use in a court of law.

Future of police AI tools

Despite the current pause, there are clear reasons for police forces to explore AI technology. Case preparation is a time-consuming administrative burden, and technology could eventually reduce these workloads. By automating some tasks, officers could potentially spend more time on other critical aspects of policing and community safety.

The Police.AI centre is currently developing specific tools designed to prepare and quality-check case files. These tools are intended to speed up charging decisions and move cases to court more quickly. However, the success of these tools depends on their ability to maintain evidence integrity without undermining the legal process.

Pilot programs for these new tools are scheduled to begin in 2026, with a national rollout planned for 2027. The goal is to balance efficiency with the high standards required by the justice system. The transition will likely be slow as authorities work to ensure that speed does not come at the expense of accuracy or justice.