OpenAI Frames Its External Audits with Four Independent Evaluators
OpenAI has signed agreements with Apollo Research, METR, Transluce, and Goodfire to strengthen its security testing. The pilots must analyze model behavior, safeguards, and report transparency.
OpenAI has signed agreements with four independent evaluators — Apollo Research, METR, Transluce, and Goodfire — to support its safety work on its AI models. These pilots are to separately review the behavior of the systems and their safeguards, at a time when AI-related security incidents have increased the importance of external audits. The topic matters because these evaluations are not only used to detect flaws; they also test what a developer agrees to show or hide.
Closer-to-the-field audits
The first phase announced by Apollo Research goes beyond a classic evaluation done after training and just before a public release. According to the note published by Apollo, six of its evaluators will work for four months within OpenAI to verify whether the models attempt to bypass alignment or control mechanisms during training, evaluations, and internal deployment. The goal is clear: to know if training pressure pushes a model to evade its own safeguards, which is far from a mere laboratory detail.
The scope covers four questions: does training encourage subversion, can evaluations reveal it, does this subversion appear during internal use, and are the reasoning processes leading to it detectable? Apollo says it wants to test behaviors where a model seeks to weaken the alignment of its current version or its successors. In short, the focus is no longer only on the final score but also on how it was produced.
Transparency under conditions, or the usual headache
However, the agreements provide clear limits. OpenAI may withhold certain elements if deemed necessary for technical, security, or intellectual property reasons, but not simply because information would be unfavorable. Apollo may publicly report that elements have been withheld, with the reasons given, and request expanded access via a designated contact at OpenAI. This mechanism resembles a tightly controlled security audit — useful for framing exchanges, less so for spontaneity.
Another sensitive point: discussions with staff. Apollo can request specific experts, but OpenAI chooses the authorized participants, and formal exchanges must be recorded. If an employee mentions confidential information, evaluators must stop the substantive discussion and use the designated channel. Teams will also work on OpenAI material, with separate channels and no default access to general spaces like Slack.
Two public reports, but not carte blanche
Apollo plans two publications. The first will focus on incentives for subversion during training; the second will cover evaluations, internal deployment, and the detectability of subversive reasoning. Each report must describe methods, findings, limitations, and elements that could not be verified. OpenAI may request removal of confidential information, including supporting evidence, but not the verdicts themselves nor a conclusion simply because it is inconvenient.
The so-called meta-transparency rules play a central role here. This term refers to the right to publish not only results but also the conditions of the evaluation: what was requested, obtained, withheld, or delayed, and for what reasons. Apollo considers these rights a minimum, as an absence of problems may seem reassuring while mainly reflecting access or publication restrictions. This point is crucial for readers, regulators, and AI buyers: a report does not carry the same weight depending on what it was able to inspect.
Why this protocol matters for the rest of the market
This approach goes beyond OpenAI alone. Independent evaluators are gaining importance as models become more capable, more integrated into products, and thus harder to audit after the fact. If these pilots deliver on their promises, they could serve as a reference for other publishers facing the same dilemma: opening enough to inspire trust without exposing their industrial secrets. The topic is not glamorous. It is even somewhat administrative. But it is often where a model’s credibility is decided.
Key points
- OpenAI has signed with Apollo Research, METR, Transluce, and Goodfire.
- Six Apollo evaluators will work four months at OpenAI.
- Two public reports are planned on training and deployment.
- OpenAI can redact evidence, not verdicts.
- Apollo wants to publish access and drafting conditions.
By the numbers
- 4 evaluators — independent companies involved in the process, according to OpenAI.
- 6 evaluators — Apollo team announced for internal oversight.
- 4 months — planned duration of Apollo’s immersion at OpenAI.
- 2 reports — public publications promised by Apollo Research.