SIAgentNews.com
← All news

OpenAI launches new image classifier and ChatGPT warnings to combat CSAM

The AI company is introducing tools to detect child sexual abuse material, including AI-generated images, while adding account suspensions for policy violations.

· First reported by TechCrunch

48%fewer images flagged by the classifier

The short version

  • A new image classifier detects CSAM, including AI-generated images, and assigns a score to help reviewers prioritize content.
  • The classifier flags 48% fewer images while finding the same number of abusive images during early partner testing.
  • ChatGPT will now show warning screens and issue 24-hour account suspensions for prompts that sexualize children.
  • OpenAI is providing law enforcement with AI-powered investigative tools, plugins, and a field guide for training.
  • The company is offering the image classifier to select companies at no cost through its API.

Why it matters

These tools represent a shift toward proactive detection of illegal content in AI systems. For users, it means stricter enforcement and automated warnings, while for developers, it provides a framework for building safer applications.

New detection for AI images

OpenAI has announced a new image classifier designed to identify child sexual abuse material (CSAM). This tool is capable of detecting both real-world images and those generated by artificial intelligence. The company developed this classifier in collaboration with the National Center for Missing and Exploited Children (NCMEC).

The classifier works by looking for visual signs of abuse in images it has never seen before. Instead of a simple yes or no, it assigns a score to the content. This score provides an indicative direction to human reviewers, helping them decide how to handle specific pieces of material.

To help spread these capabilities, OpenAI is giving select companies access to the classifier through its API at no cost. The company is also working with the Tech Coalition and ROOST to share access to the tool with other relevant organizations.

Improving reviewer efficiency

During early testing with partners, the new classifier showed significant improvements in efficiency. It flagged 48% fewer images while identifying the same number of abusive images as previous methods. This reduction in the total number of flags means that human reviewers can focus their time on more relevant content.

The data suggests that the classifier is highly accurate in its targeting. Images flagged by the new system are twice as likely to be actual abusive images compared to previous benchmarks. This precision helps streamline the process of identifying and removing harmful material from the internet.

By automating the initial screening process, OpenAI aims to make the removal of CSAM more manageable. The goal is to use AI to assist human oversight rather than replace it entirely, ensuring that high-priority content reaches the right people faster.

ChatGPT warnings and blocks

OpenAI is also updating the user experience within ChatGPT to prevent the creation of content that sexualizes children. When a user attempts a prompt related to such material, the app will now display a warning screen. This screen will direct the user toward specialized support resources to help stop harmful behavior.

In addition to the warning, the system will now temporarily suspend the user's account for 24 hours as a deterrent. While the app already blocks CSAM content and related requests, these new measures add a layer of friction for those attempting to bypass safety filters. Repeated offenses can still lead to the permanent removal of an account.

The company formed the messages and guidelines for these warnings with several organizations and researchers. These included the Lucy Faithfull Foundation and researchers from the University of Tasmania, who independently evaluated a trial of the deterrent messages. The goal is to ensure the warnings are effective and appropriate for the context.

Tools for law enforcement

OpenAI is equipping law enforcement agencies with a suite of AI-powered features to help them deal with child safety. These tools include dedicated plugins and investigative features designed for complex data analysis. Access to these specialized tools is managed by program administrators to ensure they are used correctly.

To ensure these tools are used effectively, OpenAI is providing training to investigators. They are also releasing a field guide that explains how to use AI in their specific line of work. These resources are intended to help authorities navigate the complexities of digital evidence and data analysis.

These updates come as the company faces scrutiny over the potential risks of its technology. While some groups have labeled the tech as an unacceptable risk for teens, OpenAI is attempting to build a safety blueprint. This includes updating content guidelines and retuning models to prevent the generation of harmful content.

Developer guidelines and safety

The company is also publishing new guidelines for developers who use OpenAI's tools. These guidelines are meant to help developers prevent, detect, and report CSAM content within their own applications. OpenAI stated that early testing showed these guidelines helped developers adopt child safety tooling.

The guidelines also encourage developers to register to report CSAM directly to the National Center for Missing and Exploited Children. By providing a clear framework, OpenAI hopes to create a standard for how third-party apps handle sensitive content. This move is part of a broader effort to create a safer ecosystem for AI integration.

These efforts follow a period of significant attention on how AI can be misused to create nonconsensual sexualized imagery. By providing both the technical tools and the policy framework, OpenAI is attempting to address these risks at the infrastructure level. The company continues to work with researchers to evaluate and improve these safety measures over time.

What to watch

  • How other AI companies respond to OpenAI's new classification standards.
  • Whether the 24-hour suspension period effectively deters repeat offenders.
  • The specific types of data analysis tools provided to law enforcement.

Try it

Check the National Center for Missing and Exploited Children (NCMEC) website to see how they coordinate with tech companies on reporting requirements.

Sources and links

First reported by TechCrunch. Written by the SI Agent News Desk from the reporting above.Written with AI from the source above. Every number is checked against the source. Spot a mistake? Corrections.

Liked it? Say why

Add a line for other readers.

No account needed. We never show your email. Comments with links, ads or abuse are held and not shown. Your comment shows after a quick check.

More AI news