UK Police Pause Generative AI Use in Criminal Justice Amid Hallucination and Provenance Concerns

England and Wales police forces have paused some uses of generative AI in criminal justice work, citing concerns that AI systems can generate statements that sound convincing but are false and lacking traceable provenance. The move follows incidents such as AI-generated material used by West Midlands Police in relation to a football match ban, underscoring the risk that fabricated AI outputs can appear credible. Authorities say human verification and provenance checks are essential safeguards, though these steps may reduce efficiency. Pilots from Police.AI aim to quality-check case files with a national rollout planned for 2027, balancing speed with accountability.

• Generative AI can produce plausible but untrustworthy information, raising questions about reliability in investigations and court material.
• Provenance and audit trails are critical when AI inputs influence charging decisions or outcomes in investigations.
• Human verification is deemed essential to ensure information is accurate and traceable to original evidence, even if it slows process timelines.
• Policy pilots intend to reduce administrative workload and speed up processes, but safety and accountability may dictate slower adoption.

Overview and context

Police forces in England and Wales have been instructed to pause certain uses of generative AI (AI) in criminal justice activities, including the preparation of court statements. The guidance comes in response to concerns that commercial AI systems may be deployed in situations where accuracy, provenance and accountability are essential. Alex Murray, head of the Police.AI centre, has urged forces to slow down and to assess AI tools more rigorously before they are used in criminal justice work.

Why the pause matters

The core concern is not merely that AI can err, but that generative AI can produce statements that sound plausible while being false. The risk escalates when these outputs could influence investigations or form part of evidence presented in court. The incident involving West Midlands Police, which cited information generated by Microsoft Copilot while preparing material related to a ban on Maccabi Tel Aviv supporters, illustrates the danger: Copilot referenced disorder at an earlier match that never occurred. This case demonstrates how AI-generated information can look like credible intelligence even when it is fabricated, highlighting the need for robust safeguards in criminal justice contexts.

Evidence provenance and auditability

In criminal justice, evidence requires provenance — a clear record of where information came from, how it was obtained, and why it is believed to be accurate. Generative AI complicates this: large language models (LLMs) generate outputs based on training data and inputs without verifying truth, and explanations produced by the system may themselves be part of an AI-generated record rather than a true reconstruction of the reasoning process. This raises difficult questions for defenders seeking to challenge AI-produced material, such as whether AI was used, what information was given, what was produced, whether all claims were checked against primary evidence, and whether a complete interaction log exists.

Case examples and ongoing concerns

Real cases underscore these concerns. A Derbyshire police detective has been suspended and is under criminal investigation after allegations that AI was used to create evidential material. There are also ongoing reviews of rape convictions following allegations that the detective directed an AI chatbot to produce case paperwork intended to influence charging decisions. While no charges have been brought in relation to these allegations, the case underscores the potential for AI-derived documents to influence investigations before any error is detected.

Efficiency versus safety

Despite the risks, there are potential benefits to AI in criminal justice, such as reducing time spent on administrative tasks and speeding up case preparation. Police.AI is developing tools to prepare and quality-check case files, with pilots starting in 2026 and a national rollout planned for 2027. The aim is to reduce workloads, accelerate charging decisions and bring cases to court more quickly. However, the article notes that faster preparation is only valuable if the underlying information is reliable; otherwise, AI-related errors could undermine evidence and lead to convictions being challenged or overturned.

What needs to happen next

The overarching conclusion is that human oversight cannot be a token gesture. It must involve accountability for verifying that AI-generated information is accurate, traceable and fit for use. This implies a careful balance: while AI can increase efficiency, it cannot be trusted as a substitute for human verification and document provenance in contexts where a person’s freedom is at stake. The safeguards should ensure responsibility remains with a qualified individual who can account for the information’s origins and credibility, even if that slows adoption and reduces some time savings from automation.

Conclusion

The article argues that AI can be a useful tool in criminal justice under the right safeguards, but speed cannot be the sole measure of success when the stakes include a person’s liberty. A measured, accountable approach — prioritising provenance, auditability and human verification — is essential as police forces explore AI’s potential to speed up case preparation while preserving the integrity of evidence and the fairness of the justice process.