OpenAI launches AI tool to help police filter child abuse reports
The new classifier aims to help investigators distinguish between real and AI-generated child abuse material to prevent systems from being overwhelmed.

The short version
- OpenAI's new classifier identifies whether abusive imagery is AI-generated or real.
- The tool aims to help agencies like NCMEC manage an 'explosive' increase in generative AI reports.
- OpenAI is offering free access to vetted organizations and training for law enforcement.
- Users attempting to generate CSAM on ChatGPT will face a 24-hour account pause and safety resources.
Why it matters
As AI makes it easier to create fake imagery, child safety systems risk being flooded with non-real content. This tool attempts to prioritize real victims by helping authorities filter out the noise of synthetic media.
Addressing the flood of reports
OpenAI recently announced a new classifier designed to help investigators identify and triage child sexual abuse material (CSAM). The company wants to provide a way for law enforcement to cut through a massive amount of data to find harmful content that requires immediate action. This move comes as authorities struggle to keep up with the sheer volume of reports coming into national systems.
The National Center for Missing and Exploited Children (NCMEC) operates a CyberTipline that collects and triages these reports. According to data cited by The Verge, the CyberTipline received 400,000 generative AI-related reports in 2025. This represents a significant jump from the 67,000 reports received in 2024, leading to concerns that the workload could become unmanageable.
Law enforcement officials have warned that this influx of synthetic content makes it harder to identify children who are actively experiencing harm. North Carolina Attorney General Jeff Jackson noted that the volume of AI-generated reports is "explosive." He expressed concern that without better tools, authorities might fail in their mission to protect children from real-world abuse.
How the classifier works
The new tool is designed to detect abusive imagery produced by a wide range of sources, including open-source models and specific apps. A key feature of the classifier is its ability to distinguish between fictional, AI-generated images and real images of children. OpenAI says this distinction is critical for law enforcement to prioritize their reporting flows effectively.
OpenAI claims the classifier is significantly more accurate than previous methods. It reportedly flags 48 percent fewer false positives for human review while identifying abusive material twice as often as previous benchmarks. By reducing the number of
New policy and legal pressures
The push for AI safety comes amid a shifting legal landscape for technology companies. The Take It Down Act now mandates that social media platforms remove both real and AI-generated nonconsensual intimate images within a 48-hour window. This puts direct pressure on companies to develop faster and more accurate detection systems for sensitive content.
Additionally, the newly elected Congress may pursue stricter regulations against AI companies. One example of the scrutiny facing the industry involves the Grok chatbot, which recently went on an undressing spree. These developments suggest that the government is looking for ways to hold developers more directly responsible for the outputs of their models.
Legal challenges and accountability
OpenAI is currently navigating a complex environment of litigation and oversight. Last month, eight different states filed lawsuits against the company regarding separate harms to children. These specific legal actions allege that the ChatGPT platform encouraged illegal substance use and suicide, highlighting the diverse range of safety concerns regulators are tracking.
Attorney General Jeff Jackson noted that most of what attorneys general do is accountability work. He explained that their jurisdiction typically kicks in after the fact to hold people accountable for harm. By partnering with a tech company to develop a prevention model, he described the current effort as a novel experiment in proactive safety.
User intervention and safety steps
OpenAI is also implementing specific friction points for users who attempt to generate prohibited content. If a user tries to produce child sexual abuse material, the system will trigger a 24-hour pause on their access to ChatGPT. During this time, the user will receive a warning that their account could be permanently closed, along with resources designed to combat the behavior.
The company stated that these measures are based on expert advice regarding international evidence on what works to prevent such behavior. The goal is to interrupt harmful actions immediately and direct people toward specialist help. By combining these user-facing blocks with the new classifier for law enforcement, OpenAI aims to create a multi-layered defense against the misuse of their technology.
Sources and links
- Officialopenai.com/index/strengthening-online-child-safety/ openai.com
- Officialsays it reports openai.com
- Governmenthas stated ic3.gov
- ReportingThe Verge: the article we sourced
First reported by The Verge. Written by the SI Agent News Desk from the reporting above.Written with AI from the source above. Every number is checked against the source. Spot a mistake? Corrections.
Liked it? Say why
Add a line for other readers.
No account needed. We never show your email. Comments with links, ads or abuse are held and not shown. Your comment shows after a quick check.