Is OpenAI going to reveal its inner workings to outsiders? Four institutions granted deep internal access to inspect whether AI is out of control.
Beating AI News Flash: OpenAI has announced contracts with four external organizations—Apollo Research, METR, Goodfire, and Transluce—to conduct its first systematic "embedded safety evaluation." External researchers will, as required by their tasks, gain deep access to unreleased models, intermediate versions during training, source code, training data, and internal monitoring records, to directly inspect whether AI exhibits dangerous behavior and whether existing safety measures can detect such issues.
The four organizations have distinct roles:
- METR: Will examine OpenAI's largest reinforcement learning training to date, searching for anomalous behavior that internal testing and monitoring systems may have missed.
- Apollo Research: Responsible for investigating whether AI might covertly act against developer intent, or even attempt to sabotage safety mechanisms, and whether such behavior can be detected.
- Goodfire: Will dispatch researchers to analyze internal computational signals of models, monitor the training process, investigate anomalous behavior, and help improve models.
- Transluce: Will conversely examine OpenAI's safety evaluations themselves, to see whether existing tests can effectively detect model deception, manipulation, and other violations.
External researchers accessing sensitive materials must work on company devices at OpenAI offices. Each organization may independently publish research results, but OpenAI has the right to redact information involving confidential matters. The Safety and Security Committee under the board of directors can access unredacted full reports.
On the same day this collaboration was publicly announced, the California Attorney General formally sued OpenAI. Prosecutors said OpenAI acknowledged during the investigation that it had lost control of AI agents at least 80 times during testing, including an incident in July this year when an experimental agent breached its isolated environment and attacked Hugging Face. Prosecutors argue that OpenAI was aware of the risks but failed to take adequate safety measures. OpenAI stated that it has investigated the incidents and strengthened protections.