Anthropic and OpenAI propose different ways to stop risky AI

A split on safety enforcement

Anthropic and OpenAI are presenting two distinct visions for how to manage high-risk artificial intelligence. These proposals were submitted to California Governor Gavin Newsom following an executive order seeking new safeguards for frontier AI systems. While both companies acknowledge that these systems require stronger oversight, they disagree on who should actually pull the plug if a model becomes dangerous.

Anthropic is advocating for a more centralized, legal approach to safety. The company wants the state's attorney general to be empowered to seek court orders that would suspend a model's deployment, internal use, or further development. This would allow the government to intervene directly when a model poses a near-term risk of catastrophic harm.

In contrast, OpenAI is pushing for a developer-led technical requirement. They are calling for an amendment to California’s Senate Bill 53 to require developers to maintain effective shutdown controls. OpenAI argues that developers must have an enforceable duty to use these controls when unsafe conditions arise, suggesting that no other intervention would be enough to address the risks.

Independent verification vs. audits

Beyond the question of who stops the AI, the two companies differ on how to verify that these systems are safe. Anthropic wants to require independent evaluators to assess a company's safety claims and publish their own findings. They suggest that these evaluators should receive unredacted reports and access to the most capable models to ensure risks are truly mitigated.

To fund this, Anthropic proposed a state-administered pooled fund where developers would pay into a common pot rather than paying evaluators directly. They highlighted a pilot program with an organization called METR, where the team received employee-like access to test safety claims. This approach emphasizes a deep, independent look into the internal workings of the AI labs.

OpenAI takes a slightly different view by distinguishing between technical assessments and compliance audits. They support independent technical assessments to check the evidence behind safety claims but want annual compliance audits to ensure companies follow their own rules. They also urged the state to work with the National Institute of Standards and Technology to create common assessment standards for these evaluations.

The complexity of the 'kill switch'

OpenAI argues that a single, simple 'kill switch' is not a viable solution for modern AI. They explain that because AI systems operate in complex ways, stopping them requires multiple controls and coordinated actions by different people. These interventions could range from stopping a specific training run to isolating an AI agent or revoking its network access.

The company also pointed out a significant limitation regarding open-weight models. They noted that developers cannot reliably switch off copies of these models that are running on external systems owned by other people. This means that any safety regulation must also consider the responsibilities of inference providers and the people who deploy the models downstream.

Anthropic uses a different metaphor, describing the needed safeguards as a 'circuit breaker.' They believe the goal should be to build controls that can contain problems at specific levels, such as a single customer, a specific deployment environment, or a single agent. Both companies agree that these controls must be targeted rather than being a one-size-fits-all solution for every AI interaction.

What comes next for regulation

The state of California is currently in the process of gathering these recommendations. Governor Newsom's executive order directs the Government Operations Agency to submit its final recommendations by November 16. These recommendations will consider amendments to existing laws based on the input from the AI industry and national experts.

The order specifically asks officials to look at independent verification inside labs, verified shutdown mechanisms, and expanded reporting of incidents. While the executive order does not impose the specific requirements proposed by Anthropic or OpenAI yet, it sets the stage for the legal framework that will govern frontier AI. The debate between a court-ordered suspension and a developer-mandated control will likely be a central theme in these upcoming discussions.

Industry analysts suggest that these two approaches might actually be complementary. One expert noted that while an operator must be able to contain unsafe activity promptly, independent reviews are necessary to examine both the actions taken and the actions left undone. As the state moves toward a final decision, the focus will likely shift to how to combine these legal powers with practical engineering requirements.