California Orders AI Kill Switch Policy Review
California orders an AI kill switch review under a new executive directive that accelerates independent oversight of frontier-model developers. Governor Gavin Newsom instructed state officials and outside experts to propose stronger safety rules within two months.
The September 18 order does not immediately require companies to install an emergency shutoff. It starts a policy process that could produce legislation or regulations covering verified shutdown mechanisms, onsite auditors, independently checked safety reports and a broader definition of reportable incidents.
The California AI order directs work on four safeguards:
- Independently verified kill switches for frontier models
- Onsite third-party teams conducting regular audits
- External checks of safety frameworks and risk reports
- Mandatory reporting of loss-of-control incidents
California Orders AI Kill Switch Policy Review
Newsom directed the Government Operations Agency, working with the Governor's Office of Emergency Services, to convene national experts and recommend changes to state law. The governor's official announcement says the group must deliver its guide within two months.
One proposal would require frontier-model developers to create an emergency shutoff and have an independent verification organization test its effectiveness on an ongoing basis. The language matters: California is advancing and evaluating the requirement, not claiming that a universal technical switch already exists.
A practical shutdown system would need clearly defined triggers, authorized operators and reliable control over model access, tools and computing resources. It would also need protections against misuse, because a mechanism powerful enough to stop a service can itself become a security target.
Independent Auditors Would Enter Frontier AI Labs
The review will consider requiring frontier AI companies to host a designated independent verification organization inside their laboratories. Those teams would conduct regular audits and evaluations rather than relying only on safety claims produced by the developers themselves.
California also wants safety frameworks, transparency reports and risk assessments checked against standards deemed adequate by an independent verifier. That would move oversight closer to financial auditing, where an external party evaluates evidence and controls instead of merely receiving a company's public summary.
The order accelerates implementation of two laws Newsom signed earlier in September. SB 813 establishes a framework for certifying independent AI verification organizations, while AB 1405 creates a state registry for AI auditors and standards covering their independence, transparency and integrity.
Loss-of-Control Incidents Could Become Reportable
California's existing SB 53 framework already requires covered frontier developers to publish safety frameworks, report specified critical safety incidents and protect whistleblowers. The new order asks experts to consider expanding the definition of a critical incident to include cases in which an AI system loses control of its operations.
The state announcement points to a recent attack involving Hugging Face as the kind of episode that could shape the expanded definition. A broader reporting rule could give regulators earlier visibility into autonomous behavior, unauthorized access, unexpected tool use or failures to obey operational limits.
Reporting definitions will determine whether the policy produces useful evidence or a flood of ambiguous notices. Regulators must distinguish serious loss of control from routine model errors, unsuccessful tests and low-impact software failures while still capturing near misses that reveal systemic weaknesses.
The Verge reported that the proposals include regular onsite audits, independent verification of required filings and ongoing checks of any shutdown mechanism. StateScoop reported that recommendations are due November 16.
California Builds on SB 53, SB 813 and AB 1405
The order connects three layers of California policy. SB 53 created disclosure and incident-reporting duties for frontier developers. SB 813 provides the foundation for certifying outside verification organizations. AB 1405 supplies a registry and professional standards for auditors assessing compliance with state law.
The next stage is converting those structures into operating rules. Officials must decide which models qualify as frontier systems, what evidence auditors can inspect, how conflicts of interest are prevented, and whether state agencies can enforce a shutdown requirement across cloud providers and model distributors.
California is also asking Washington to treat its framework as a national baseline. A federal standard could reduce conflicting state obligations for developers, but Congress has not enacted a comprehensive frontier-AI safety regime. That leaves California using its large technology sector and regulatory reach to set an early template.
The November recommendations will show whether the kill-switch language becomes a technically detailed mandate or remains a policy objective. The most credible proposal will specify who can activate it, what systems it reaches, how independent testers verify it and how companies report a failure without creating a new attack surface.
