SIAgentNews.com
← All news
DevelopingOpenAISecurity

OpenAI limits hidden reasoning extraction to under 1 percent

The company is deploying new defenses to prevent users from stealing the internal logic used by its most advanced models to solve complex tasks.

· First reported by Superpower Daily

Ink and watercolour: picture for the story 'OpenAI limits hidden reasoning extraction to under 1 percent'
AI-made ink and watercolour for this story. No real people or products are shown.
below 1%success rate of hidden-reasoning extraction

The short version

  • OpenAI reports that adaptive, black-box attacks to extract reasoning from GPT-6.2 Astra succeeded less than 1% of the time.
  • The company is using a mix of API controls, automated checks, and account-level enforcement to block unauthorized data copying.
  • Research shows that exact reasoning traces can transfer more cyber capabilities to student models than simple summaries.
  • OpenAI reports taking action against thousands to tens of thousands of accounts weekly to disrupt distillation campaigns.

Why it matters

This highlights a growing arms race between AI developers and those trying to 'distill' high-end capabilities into smaller, cheaper models. If successful, these attacks allow others to copy advanced skills while bypassing the safety rules built into the original model.

The challenge of hidden reasoning

OpenAI recently released a security report detailing efforts to protect the internal logic of its models. This internal logic, often called chain of thought, consists of the intermediate steps an AI takes to reach a final answer. These steps are often hidden from the user but contain significant information about how the model functions.

The company is particularly concerned about a process called distillation. This occurs when a developer trains a smaller, less capable model using the outputs or internal processes of a much more powerful model. By doing this, they can essentially copy the advanced capabilities of a frontier model into a cheaper version.

A major risk of this process is that the distilled model might inherit the skills of the original without inheriting its safety rules. OpenAI argues that protecting these reasoning traces is essential to ensure that advanced AI capabilities are not copied in ways that could lead to misuse. As Superpower Daily reported, the company is actively working to limit this unauthorized copying.

Testing against adaptive attackers

To test their defenses, OpenAI used automated AI attackers that behaved like external users. These were black-box tests, meaning the attackers had no special access to the internal systems and could only see what a normal user would see. The attackers were adaptive, meaning they adjusted their tactics based on the responses they received from the defender.

The company tested these attacks against GPT-6.2 Astra to see if they could recover reasoning from both current requests and earlier ones. The results showed that the new defenses brought the success rate of these extraction attempts to below 1% in internal tests. While the report notes that the comparison involves different model generations, it highlights a significant reduction in successful extractions.

The testing covered six different attacker models to ensure a broad range of methods were accounted for. By simulating these sophisticated, evolving attacks, OpenAI can better understand how malicious actors might try to probe their systems. The goal is to create a robust barrier that holds up even against non-static, intelligent adversaries.

The value of reasoning traces

A separate experiment within the report looked at how much information actually transfers during distillation. Researchers compared training a student model on exact reasoning traces versus training it on simple summaries of tasks. They specifically looked at coding and defensive cybersecurity tasks to see which method produced a more capable student model.

The findings suggested that exact reasoning traces generally outperformed summaries when the student model had a large computation budget. However, the summary-trained student performed better when the computation budget was lower. This indicates that high-quality reasoning data is a more potent way to transfer complex skills than summarized information.

OpenAI interprets these findings as evidence that even reasoning from permitted defensive work can teach a model skills that extend beyond the original task. This reinforces the need for strict controls on reasoning data. If a student model can learn complex cyber capabilities from a few reasoning traces, the value of that data to unauthorized users is very high.

Multi-layered defense strategies

OpenAI is not relying on a single fix but is instead using a combination of different security layers. These include specific training methods that teach models to refuse extraction attempts while still providing helpful explanations to legitimate users. They also use automated checks to inspect the messages and arguments sent to tools.

These automated checks are designed to catch reasoning that might leak without appearing in the final answer. If a suspected leak is identified, the system can block it before it ever reaches the user. This happens at the request level, providing a first line of defense against real-time probing.

Beyond individual requests, the company monitors for broader patterns across different accounts. They identify networks that violate their rules and enforce actions against them. OpenAI reports taking action against campaigns involving thousands to tens of thousands of accounts every week to reduce the volume and value of training data obtained by unauthorized distillers.

Future monitoring and goals

The company's primary goal is to reduce the amount and value of training data that unauthorized distillers can obtain. They are tracking how prevalent abuse is and how quickly their detection systems can lead to enforcement actions. These investigations provide a feedback loop that informs future training and testing cycles.

OpenAI is also applying some of these selected safeguard improvements to its GPT-6 Astra model. By constantly updating the defenses, they aim to stay ahead of attackers who are looking for ways to bypass restrictions. The report acknowledges that more study is needed across different student models and training methods to fully understand the scope of the problem.

As the AI landscape evolves, the tension between model transparency and security will remain a central theme. OpenAI's approach suggests a move toward

Sources and links

Read the original at Superpower Daily

First reported by Superpower Daily. Written by the SI Agent News Desk from the reporting above.Written with AI from the source above. Every number is checked against the source. Spot a mistake? Corrections.

Liked it? Say why

Add a line for other readers.

No account needed. We never show your email. Comments with links, ads or abuse are held and not shown. Your comment shows after a quick check.

More AI news