Anthropic releases new standards for testing AI safety

Moving beyond simple refusals

Anthropic recently published a set of recommendations for conducting more reliable AI safety evaluations. The company argues that current methods, which often rely on single-turn API tests, can provide a misleading sense of security. While these tests show high harmless-response rates, they are primarily useful for checking known failure modes rather than predicting real-world behavior.

A major issue identified is 'construct validity,' which refers to whether a test actually measures the concept it claims to. For example, the company notes that a high refusal rate is not always a good thing in sensitive scenarios like self-harm. In those cases, simply declining to engage with a user can be a failure because the goal is to provide support and direct the user toward human help.

To address this, Anthropic suggests measuring an 'appropriate-response rate' across an entire conversation. This approach uses specific rubrics where a single bad turn can cause the entire exchange to fail. By looking at the flow of a conversation, researchers can better understand if a model maintains its safety boundaries over time.

How multi-turn tests work

To create these complex evaluations, domain experts write detailed specifications for a simulated user. These specifications include the user's objectives, tactics, persona, tone, and specific situation. This setup allows for a more realistic simulation of how a human might interact with an AI in a high-stakes scenario.

The testing process involves a secondary Claude model that generates the user's responses based on the expert's instructions. This allows the tested model to interact with a variety of different conversational styles and tactics. By varying these dimensions independently, researchers can identify exactly what drives a model's behavior, such as whether it reacts more strongly to an urgent tone or the severity of a request.

Anthropic also advises building these tests from real-world data rather than just invented situations. They recommend using public data, red-teaming transcripts, incident reports, and expert interviews. This ensures that the safety evaluations are grounded in the types of prompts and behaviors that users actually encounter in the wild.

The role of deployment context

One of the most significant findings in the new guidance is that a model's safety is tied to its deployment configuration. A model might behave safely in one environment but fail in another depending on the system prompts provided by the developer. The company demonstrated this by showing how different settings could lead to vastly different results for the same model.

In one example, a specific model version showed a 94% appropriate-response rate on a consumer website but only a 60% rate on its API. This discrepancy occurred because the website used targeted system-prompt mitigations that were absent in the API version. This proves that a headline safety score cannot be treated as a universal truth if the deployment settings are not identical.

Because of these variables, Anthropic recommends that developers report their specific sampling settings, prompts, and grader versions. They also suggest continuous evaluation during the fine-tuning process rather than performing a single check before release. This ensures that changes made during development do not introduce new, unforeseen safety risks to the final product.

Validating automated graders

The guidance acknowledges that using automated 'grader models' to score AI behavior introduces its own set of risks. If a grading system is flawed, it might continue to produce plausible-looking scores that do not reflect reality. To counter this, Anthropic suggests using atomic yes-or-no criteria and explicit rules for handling borderline cases or disclaimers.

To ensure accuracy, human reviewers should independently label a sample of transcripts to establish a baseline. The company suggests comparing the agreement between humans with the agreement between the grader and the humans. If experts agree with each other but not with the grader, it signals that the rubric or the automated system needs to be adjusted.

Finally, the report emphasizes the importance of statistical power and transparency. Anthropic suggests using floors of 100 items for complex multi-turn tests and 1,000 for simpler ones to account for noisy outputs. They also advocate for preregistering metrics and improvement thresholds before results are seen to prevent biased reporting of safety improvements.