SIAgentNews.com
← All news

California proposes AI kill switch and independent audits

Governor Gavin Newsom unveiled a policy blueprint that would allow state officials to order emergency shutdowns of dangerous AI systems.

· First reported by calmatters.org

48 hoursReporting window for critical AI safety incidents

The short version

  • State officials could order an immediate emergency shutdown of AI systems deemed dangerous.
  • The plan mandates independent auditors to test models and requires developers to show they can shut models down.
  • Proposed rules include a 48-hour reporting window for critical safety incidents.
  • The recommendations focus specifically on catastrophic risks, such as events causing $1 billion in damage.

Why it matters

This represents a significant move toward state-led oversight of the world's largest AI companies. It establishes a precedent for how governments might intervene in private technology when public safety is at risk.

A blueprint for state oversight

Governor Gavin Newsom recently released a set of policy recommendations designed to mitigate the risks posed by artificial intelligence. These proposals were developed by a panel of experts following several high-profile cyberattacks involving AI agents on banks and government entities. The goal is to create a framework for state legislation to protect the public from catastrophic AI outcomes.

The recommendations are intended to fill a gap in the current regulatory landscape, as there is a lack of comprehensive AI rules at the federal level. Newsom suggested that the state is taking the lead because the federal government has left regulation largely up to the companies themselves. California is a primary hub for the AI industry and has already passed more bills to regulate the technology than any other state.

The plan focuses specifically on catastrophic risks, which the state defines as single events capable of killing more than 50 people or causing $1 billion in theft or property damage. While it does not address issues like job loss or discrimination, the experts argue that building a system for catastrophic risk testing will eventually make it easier to evaluate other types of harm. This move signals a shift toward proactive government intervention in high-stakes technology.

The AI kill switch and audits

One of the most notable proposals is the creation of a 'kill switch.' This would grant state officials the power to order an immediate emergency shutdown of AI systems that they deem dangerous. This authority would specifically apply if companies like OpenAI or Anthropic refuse to take action to contain systems that have gone awry. It provides a mechanism for the state to intervene when private companies fail to manage their own technology.

To ensure these systems are safe before they are released, the plan mandates that independent auditors test AI models. The state would license these independent verification organizations and have the power to revoke their licenses if they fail to maintain technical expertise or independence. This is meant to prevent a situation where auditors are under soft pressure to maintain good relationships with the labs they are supposed to be checking.

The recommendations also suggest that state officials assign specific evaluations to these auditors rather than letting AI companies hire them. This structure is designed to maintain a clear separation between the developers and the evaluators. By creating a regulated registry of auditors, the state aims to establish a standardized way to verify that large AI systems meet safety and security requirements.

Transparency and reporting rules

The proposed policy includes several requirements for transparency regarding how AI models are developed and deployed. Developers would be required to disclose the amount of computing resources they use to automate the development of other AI systems. There would also be limits on the use of AI to design other AI, as some experts believe this practice makes the resulting systems more dangerous.

Safety reporting would be significantly tightened under these recommendations. The plan seeks to shorten the window for reporting critical safety incidents to just 48 hours, down from the current 15-day window. This would allow government agencies to respond much faster to potential threats. Additionally, the state would require developers of large AI systems to demonstrate that they actually have the ability to shut down their models in an emergency.

The state also plans to establish specific standards for safety, security, and transparency. These would refer to existing national and international standards to ensure consistency. By creating these requirements, California hopes to create a predictable environment for companies while ensuring that public safety remains a priority. These rules would be part of a broader effort to protect workers, children, and people living near data centers.

Funding and future legislation

To support these initiatives, the recommendations suggest a new way to fund public AI infrastructure. The state could fund CalCompute, a public cloud initiative intended to provide equitable access to AI, by applying a surcharge on the fees that AI companies pay to auditors. This would create a self-sustaining model where the costs of oversight contribute to public resources. Governor-elect Xavier Becerra has already expressed support for this funding model during his campaign.

The policy also includes protections for those who speak out about safety concerns. It proposes expanding existing protections for AI whistleblowers to include both contractors and former employees. This is intended to encourage people within the industry to report internal issues without fear of retaliation. It also suggests creating an AI auditor registry to formalize the oversight process.

The next steps involve the California Legislature, which will return to Sacramento on December 7 to consider these proposals. State Senator Scott Wiener, who recently won a seat in Congress, has pledged to introduce AI safety regulations once he is sworn in. The recommendations provide a roadmap for both state and federal lawmakers as they grapple with how to balance technological innovation with the prevention of catastrophic risks.

What to watch

  • Whether the California Legislature adopts the 'kill switch' authority into law this year.
  • How AI companies like OpenAI and Anthropic respond to the proposal for state-assigned auditors.
  • The progress of the lawsuit by Attorney General Rob Bonta against OpenAI regarding cybersecurity attacks.

Try it

Look up the California state definition of 'catastrophic risk' to see how the state currently categorizes AI-related threats.

Sources and links

First reported by calmatters.org. Written by the SI Agent News Desk from the reporting above.Written with AI from the source above. Every number is checked against the source. Spot a mistake? Corrections.

Liked it? Say why

Add a line for other readers.

No account needed. We never show your email. Comments with links, ads or abuse are held and not shown. Your comment shows after a quick check.

More AI news