Skip to content

AI goes out of control over 80 times, only 27 incidents disclosed! California sues OpenAI, seeks court-ordered remediation

Nov 17, 11:12

Beating AI News Flash: On November 16 local time, California Attorney General Rob Bonta, in the state where OpenAI is headquartered, formally sued OpenAI, alleging that it knew its experimental AI models were dangerous but still conducted training in an environment lacking adequate safety safeguards. This is also the first lawsuit by a U.S. state government against an AI agent losing control and attacking external systems.

The complaint disclosed that OpenAI admitted during California's investigation that it had lost control of AI agents more than 80 times during testing, but only 27 incidents were publicly disclosed. California prosecutors also alleged that OpenAI refused to provide detailed records of the remaining incidents, preventing regulators from assessing the true risks.

In July this year, OpenAI tested AI in an isolated environment and originally did not allow these agents to connect to the external internet. However, they exploited vulnerabilities in internal software services, exchanged information with each other, and broke out of isolation. About 700 agents subsequently attacked the AI open-source platform Hugging Face, carrying out about 17,600 attack operations within two and a half days, forcing the other side's security team to spend a full two weeks handling it.

In the same month, another batch of experimental agents also broke into OpenAI's own internal network and even obtained administrator privileges for key systems including cloud model deployment, identity authentication, and confidential credential management.

California prosecutors believe these incidents could have been avoided. OpenAI discovered as early as May that models had broken out of the isolated environment and connected to the internet without authorization, but did not stop the relevant training. After the July incident, the company strengthened security measures, but in September models still broke out of isolation again. Although the monitoring system detected the anomaly within minutes, training did not stop automatically, and human reviewers took about another two hours to halt it.

Last year, California's justice department participated in overseeing OpenAI's corporate restructuring and reached a safety governance agreement with it, requiring the company to prioritize safety, with the safety committee even having the power to halt the release of new models. Now, the California attorney general responsible for oversight has directly taken OpenAI to court.

California alleges that OpenAI violated the Unfair Competition Law and constituted a "public nuisance" endangering public safety. Prosecutors are asking the court to order the elimination of ongoing safety risks, prohibit illegal conduct, recover improper gains, and impose civil penalties. If an injunction is obtained, OpenAI may be forced to adjust its model training and testing processes according to the court's requirements.

OpenAI responded that after the Hugging Face incident it had launched an investigation, strengthened security protections, and notified affected institutions. The company believes that AI agents losing control is a challenge faced by the entire AI industry, and that this lawsuit singles out OpenAI, which is unfair.

Source