AI tool that screens draft laws for corruption risk finds under half of expert flags
A new arXiv paper tests an AI assistant that flags corruption-prone wording in bills, and its honest scores show how far the idea still has to go.
The short version
- The assistant flags bill provisions an expert would criticise and names the regulatory defect in each.
- Test set: 132 draft laws and 474 findings from Ukraine's anti-corruption agency, NAZK.
- Top ten flags matched 0.28 of findings with Gemma 4 31B and 0.30 with Sonnet 5.5.
- None of the 32 findings about conflicts between laws was matched.
- Scores come from an AI judge, not human experts.
Why it matters
Checking bills for corruption risk is slow expert work, so a screening aid could help overstretched reviewers. This paper shows it is a first-pass helper at best: it misses more than half of what the agency flagged, and its scores are unverified by human experts. Anyone tempted to trust such tools on real legislation should read those limits first.
What the researchers built
As arxiv.org reported, the paper describes an assistant that reads a draft law and points to provisions an expert reviewer would criticise. For each one, it names the kind of regulatory defect involved. It is built on a framework called TRACE.
The system is a pipeline, not a single chatbot prompt. A fixed, rule-based step first splits the bill into provisions. A language model then judges one provision at a time, rules assign risk factors, and humans oversee the process from outside.
How it was tested
The team built an open benchmark from published conclusions of Ukraine's National Agency on Corruption Prevention (NAZK). It covers 132 draft laws and 474 findings, which were extracted by a model rather than by hand.
A flag counts as a match if it targets a compatible provision and an AI judge rates the risk as the same (strict) or partly the same (lenient).
What the scores show
On 100 draft laws, the top ten flags matched 0.28 of findings using the local Gemma 4 31B model and 0.30 using Sonnet 5.5. Counting all flags, the match rate rose to 0.43-0.46. That run was not blind, and the frozen blind baseline scored 0.21.
Most misses were not failures to find the right passage. The system located the provision but labelled it as a different risk. None of the 32 findings about legal collisions, where a bill conflicts with other law, was matched.
On 21 acts held out from tuning, the pipeline beat a single long prompt containing the whole methodology under the strict measure, but not the lenient one. Opus 5.5 added nothing over Sonnet.
The caveat that matters
The authors say the scores reflect agreement with model-extracted findings, rated by an AI judge, without validation by human experts. The judge's two access routes agreed on 73 of 74 labels, which shows consistency, not correctness.
So the numbers describe how closely the tool mimics one agency's written conclusions, as filtered through AI, not how well it catches real corruption risk.
What to watch
- Will human legal experts validate the benchmark and the judge's ratings?
- Can the system learn to catch conflicts between laws, where it matched none of 32?
- Does it work on legal systems beyond Ukraine's?
Sources and links
- Reportingarxiv.org: the article we sourced
First reported by arxiv.org. Written by the SI Agent News Desk from the reporting above.Written with AI from the source above. Every number is checked against the source. Spot a mistake? Corrections.
Liked it? Say why
Add a line for other readers.
No account needed. We never show your email. Comments with links, ads or abuse are held and not shown. Your comment shows after a quick check.