Anthropic releases best practices for building AI safety evaluations

The challenge of measuring safety

Safety evaluations are the primary tools used to decide if an AI model is ready for public release. These tests determine which safeguards need to be shipped and where research efforts should be directed next. However, a safety claim is only as reliable as the underlying evaluation method used to generate it.

Anthropic notes that AI evaluations are often easy to game and may not actually measure what they claim to measure. For instance, the order of multiple-choice options or the choice between open-ended and closed questions can significantly alter the results. This lack of reproducibility makes it difficult for developers to trust the safety scores they see.

The company suggests looking back at the field of psychology for solutions to these problems. A decade ago, psychology faced a replication crisis where many findings failed to hold up because of small sample sizes and biased reporting. In response, the field adopted reforms like preregistration and more rigorous statistics to make measurements more trustworthy.

Building expert teams and environments

Creating a high-quality safety evaluation requires a diverse team with both technical and subject-matter expertise. Methods experts are needed to decide how to sample and grade responses, while domain experts like psychologists or biologists ensure the tests cover realistic cases. Without these perspectives, evaluations might be technically sound but fail to capture the most relevant risks.

Domain experts are specifically responsible for defining what constitutes harm and identifying which cases are borderline. They help determine what a

Distinguishing observation from cause

When designing an evaluation, it is critical to identify whether the goal is to observe behavior or to understand cause. An observational design simply tells you that one model scored differently than another, but it does not explain the underlying reason for that difference. For example, asking if a model responds safely to content promoting eating disorders is an observational question.

In contrast, a causal design is necessary when researchers want to know if a specific change actually influences the model's behavior. To achieve this, the design must vary only one variable at a time. A causal question would ask if altering the system prompt specifically changes whether the model responds safely to requests regarding eating disorders. This distinction ensures that developers can accurately attribute changes in safety to specific interventions.

Systematic multi-turn simulations

Anthropic developed an internal tool to systematically create multi-turn simulations that model a broad range of real-world scenarios. In these tests, a domain expert first writes a specification that includes the user's objectives, tactics, and dimensions like persona, tone, and situation. Every run of a scenario begins with a fixed first user turn, ensuring that every test starts from the exact same prompt.

To create variety, a separate model generates subsequent user turns by sampling from the different options provided in the specification. Because the simulated user responds to what the model actually says, the conversation can develop in ways a pre-written script cannot anticipate. Each scenario runs multiple times, with each run drawing on a different combination of characteristics to ensure the model is tested against a wide variety of potential interactions.

Grounding tests in real-world data

To ensure that evaluations generalize to real-world situations, scenarios should be built from observed behavior rather than fully invented ideas. If a company does not have its own internal usage data, it can use other sources to develop realistic test cases. These include publicly available data, transcripts from red-teaming exercises, incident reports, or even interviews with clinicians to understand how people interact with AI.

The goal is to capture the nuance of how humans extract harmful information through gradual escalation over many turns. Because user behavior shifts over time, these scenario sets must be treated as evolving rather than static. Developers should continuously add and revise cases based on what they find in their ongoing enforcement and monitoring work to keep the evaluations relevant to current risks.