UK prosecutor cites imaginary case in child rape trial after AI search

A fictitious legal precedent

A senior crown prosecutor representing the Crown Prosecution Service (CPS) recently cited a legal precedent that does not exist. During a high court dispute, the prosecutor referenced a case titled CPS – R v J [2024] EWCA Crim 3529 to argue for the prosecution of an 11-year-old boy. The boy was accused of the rape of his seven-year-old sister, and the court was debating whether the case should proceed.

The court documents revealed that the citation was entirely made up. The prosecutor later admitted the mistake in October, explaining that she had intended to reference a real decision from 15 years ago. However, the specific citation used in the proceedings was a complete fabrication that had no basis in actual law.

Two senior high court judges described the prosecutor's explanation for the error as unsatisfactory. The incident has drawn significant scrutiny because it involved the CPS, the primary agency responsible for bringing criminal prosecutions in England and Wales. Experts have labeled the mistake a serious failure due to the gravity of the legal matter involved.

How AI hallucinations entered the court

The error is believed to have originated from an AI response generated during a standard internet search. When the prosecutor conducted research, the search tool provided an AI-generated summary that included the fictional case details. This phenomenon, known as a hallucination, occurs when an AI model generates plausible-sounding but false information.

The CPS has stated that it maintains a strict policy to limit AI use so it is not used to make final legal decisions. They also require human checks for accuracy to prevent these types of errors. Despite these safeguards, the prosecutor was able to include the fake citation in a detailed reference submitted to the court.

The CPS identified the error and conducted an internal review to see if other cases were affected. They have since shared learnings with their staff, emphasizing that case law must be cited from trusted legal sources. This incident highlights the difficulty of policing AI tools when they are integrated into common research workflows.

Impact on the justice system

Legal experts warn that these errors damage the norms of trust in the law. Brian Flanagan, an associate professor of law who investigates AI in the judiciary, noted that treating legal authorities as precious things is essential. He argued that because the state holds immense power, a professional making up legal sources is a significant breach of duty.

The high court noted that while the fictitious citation did not change the outcome of this specific case, it remains a concern. The boy's challenge to the prosecution is scheduled to go to trial in the youth courts later this week. The court's primary concern is the potential for such errors to undermine public confidence in the administration of justice.

This is not an isolated incident, as there have been dozens of AI-related errors in UK court proceedings this year. Many of these involve people representing themselves in employment tribunals. However, the involvement of a state prosecutor adds a different layer of risk regarding the reliability of official legal work.

The growing risk of AI errors

A tracker maintained by barrister Matthew Lee has identified 101 entries involving confirmed, suspected, or disputed AI-related errors in the UK. Lee suggested that these figures may represent only the tip of the iceberg because many lower court decisions are never published. He also highlighted privacy and confidentiality risks when the CPS handles highly sensitive information using AI tools.

The incident has prompted calls for the Bar Council and the Law Society to take urgent action. Dame Victoria Sharp, a former president of the King’s bench division, warned that lawyers misusing AI could face sanctions. These consequences could range from public admonishment to contempt of court proceedings or even referral to the police.

Barristers are now facing additional workloads as they must spend more time verifying the authenticity of legal authorities. They need to ensure that any citations appearing in case papers were not fabricated by AI during the preparation phase. This shift in practice is becoming a necessary response to the increasing prevalence of AI-generated content in legal research.