SIAgentNews.com
← All news

California proposes an AI kill switch to halt dangerous systems

Governor Newsom released a policy blueprint that would allow state officials to order emergency shutdowns of AI models that pose catastrophic risks.

· First reported by KQED

Documentary photo: picture for the story 'California proposes an AI kill switch to halt dangerous systems'
AI-made documentary photo for this story. No real people or products are shown.
48 hoursNew reporting window for critical safety incidents

The short version

  • State officials could order an immediate emergency shutdown of AI systems deemed dangerous.
  • Independent auditors would be licensed by the state and potentially assigned by the government.
  • Developers would be required to disclose computing resources used to automate AI development.
  • The proposal targets catastrophic risks, such as cyberattacks causing over $1 billion in damage.
  • Reporting windows for critical safety incidents would be shortened from 15 days to 48 hours.

Why it matters

This represents a shift toward active government intervention in AI safety rather than relying on voluntary corporate guidelines. For the public, it suggests a future where the state has the legal authority to pull the plug on technology that threatens infrastructure or safety.

The blueprint for a kill switch

Governor Gavin Newsom recently unveiled a comprehensive set of policy recommendations designed to manage the risks of artificial intelligence. A central feature of this plan is the creation of a "kill switch," which would grant state officials the power to order an immediate emergency shutdown of AI systems. This authority would specifically apply when companies like OpenAI or Anthropic fail to contain systems that begin to behave dangerously.

The recommendations emerged from a panel of experts assembled by Newsom following several high-profile cyberattacks. These attacks targeted various entities, including banks, private companies, and government agencies, using AI agents. The governor intends to use this report as a blueprint for new state legislation to fill a perceived lack of federal oversight.

Newsom framed the move as a necessary step to protect the public from the risks of rapid technological advancement. He argued that the state is stepping in because the federal government has left AI regulation largely up to the companies themselves. The plan focuses specifically on mitigating catastrophic risks rather than addressing broader issues like job loss or discrimination.

Mandating independent oversight

A major pillar of the proposal involves the creation of a formal registry for independent verification organizations, known as auditors. The state would have the power to license these auditors and revoke their credentials if they fail to maintain technical expertise or independence. This is intended to prevent a situation where AI labs only hire auditors who are willing to overlook safety flaws to maintain good business relationships.

To further ensure objectivity, the recommendations suggest that state officials should assign specific evaluations to these auditors. This would move away from a model where AI companies choose their own evaluators. The goal is to create a standardized system where the state can verify that models meet safety and security benchmarks before they are deployed widely.

The plan also includes requirements for developers to demonstrate that they have the technical ability to shut down their models in an emergency. Furthermore, it suggests that the state should limit the use of AI to design other AI systems. Experts on the panel believe that using AI to automate the development of more complex systems could make those systems significantly more dangerous and harder to control.

Transparency and reporting rules

The policy recommendations include several new transparency requirements for large AI developers. Companies would be required to disclose the amount of computing resources they use to automate the development of AI. This data would help the state monitor the scale of development and the resources being poured into increasingly powerful models.

The proposal also seeks to tighten the timeline for reporting safety incidents to government agencies. Currently, the window for reporting critical incidents is 15 days, but the new recommendations would shorten that window to just 48 hours. This faster reporting is intended to give regulators more time to react to potential threats before they escalate into larger problems.

Additionally, the plan calls for expanding protections for AI whistleblowers to include both former employees and contractors. By protecting those who speak up about internal safety concerns, the state hopes to create a more transparent environment. These measures are part of a broader effort to establish state standards for safety, security, and transparency that align with international norms.

Funding and legislative path

To support these initiatives, the report suggests a new funding model for CalCompute, a public cloud initiative designed to provide equitable access to AI. The plan proposes a surcharge on the fees that AI companies pay to auditors to fund this public resource. This would create a system where the costs of safety oversight contribute to a public infrastructure for AI development.

The recommendations come at a time of significant legislative activity in California, which has passed more AI-related bills than any other state. Governor Newsom recently signed dozens of bills into law to protect various groups from AI impacts, including children and workers. The state is also moving to accelerate the implementation of laws that create the auditor registry and set standards for those auditors.

Legislators are expected to return to Sacramento on December 7 to consider these new proposals. State Senator Scott Wiener, who recently won a seat in Congress, has already pledged to introduce AI safety regulations. While the current report focuses on catastrophic risks, the authors argue that building a system for credentialed auditors to test for these risks will eventually make it easier to evaluate models for other types of harm.

What to watch

  • Whether the California Legislature adopts the 'kill switch' authority into law by the end of the year.
  • How AI companies like OpenAI and Anthropic respond to the proposal of state-assigned auditors.
  • The progress of the lawsuit by Attorney General Rob Bonta against OpenAI regarding cybersecurity attacks.

Try it

Look up California's current definition of 'catastrophic risk' to see how the state legally classifies cyberattacks and autonomous system damage.

Sources and links

Read the original at KQED

First reported by KQED. Written by the SI Agent News Desk from the reporting above.Written with AI from the source above. Every number is checked against the source. Spot a mistake? Corrections.

Liked it? Say why

Add a line for other readers.

No account needed. We never show your email. Comments with links, ads or abuse are held and not shown. Your comment shows after a quick check.

More AI news