Anthropic urges California to allow court orders to block unsafe AI

A call for regulatory intervention

Anthropic has formally requested that California Governor Gavin Newsom grant government officials the authority to seek court orders against artificial intelligence models. This request specifically targets systems that are identified as posing serious risks to the general public. The company wants a mechanism where regulators can step in when safety reviews flag a model as a significant threat.

This move comes at a time when United States policymakers are actively debating how to create stricter rules for advanced AI. The goal is to establish a clear path for preventing harm before it occurs on a large scale. Anthropic's proposal aims to provide a legal framework for these types of emergency interventions.

The company submitted these recommendations following a September meeting where Governor Newsom convened a panel of AI experts. These experts were tasked with developing specific proposals for safety requirements in the state. Anthropic's suggestions are expected to influence how California handles AI oversight, which may serve as a blueprint for other states.

Independent oversight and reporting

A core part of Anthropic's proposal involves the use of independent safety evaluations to monitor the AI landscape. The company suggests that external evaluators should have a direct line of communication to California's attorney general. This would allow for the reporting of serious safety concerns without going through the developer first.

To ensure transparency, Anthropic also supports a requirement for large AI developers to publish risk reports on a regular basis. Specifically, the company recommends that these reports be released at least every six months. This consistent reporting would help regulators identify potential problems before they result in significant damage to the public.

By creating a system of independent scrutiny, the company believes regulators can better understand the risks of advanced systems. This approach moves away from relying solely on a company's internal self-reporting. It establishes a more rigorous standard for how the safety of high-powered models is verified and communicated to the government.

Targeted measures vs. full shutdowns

While Anthropic supports the idea of emergency controls, it has expressed specific concerns about rules that would require companies to build 'kill switches' for entire models. The company argues that a full shutdown of a system should be reserved for extreme situations only. They believe that such a broad measure might be unnecessary for many types of malfunctions.

Instead of a total shutdown, Anthropic favors more targeted interventions to address specific problems as they arise. Examples of these measures include disabling a malfunctioning AI agent or restricting access for a single customer. These actions would allow for the resolution of a specific issue without taking an entire technological system offline.

The company's stance is that continuous monitoring is more effective than a binary choice between full operation and total shutdown. By focusing on specific problems where they occur, developers and regulators can maintain the utility of the technology. This approach seeks to balance the need for safety with the desire to keep useful AI tools available to the public.

The broader legislative landscape

Anthropic's recommendations join a wider national debate regarding the government's authority to block the release of unsafe AI models. In September, a bipartisan group of senators discussed legislation aimed at establishing these types of rules. However, that specific proposal has not yet advanced through the legislative process due to the pre-election break in Congress.

The US House of Representatives has also considered the AI Kill Switch Act. This piece of legislation would give officials the power to order companies to shut down systems that threaten the economy or human life. Anthropic's current proposal provides a more nuanced view of how these powers might be exercised in practice.

The challenge for policymakers remains finding a way to establish clear standards for intervention. They must do this without creating rules that unnecessarily restrict the development of helpful AI technologies. Anthropic's focus on independent reports and targeted controls offers one potential path forward in this complex debate.

Impact on the AI industry

The proposed rules could significantly change how large AI developers operate within the state of California. Companies would need to prepare for more frequent reporting and potentially more rigorous external audits. This could lead to a more standardized safety protocol across the industry as California sets its policy.

For developers, the shift toward targeted interventions might be more manageable than building infrastructure for total system shutdowns. It allows for a more surgical approach to safety, where a single bad actor or a specific bug can be neutralized. This could preserve the integrity of the broader model while still addressing immediate risks.

Ultimately, these recommendations highlight the tension between technological progress and public safety. As the conversation moves from theory to potential law, the specific details of these court orders will be critical. They will determine how much power the government truly has to halt the development of the next generation of AI.