OpenAI automates debugging of ChatGPT slowdowns
OpenAI has developed Performance Factory, an internal system based on Codex to detect, diagnose, and track performance regressions. The tool handled over 1,700 investigations in just over three months.
OpenAI has developed Performance Factory, an internal system powered by Codex that automates software slowdown investigations and tracks performance regressions. The company is already using it to keep ChatGPT fast, as code agents multiply changes and complicate manual review. According to an engineering report shared with VentureBeat, the tool conducted more than 1,700 investigations and identified the root cause in over 600 cases in just over three months.
Quand le débogage devient un travail d’agent
The initial problem is quite simple: when an update slows down an application, engineers must sift through metrics, trace requests between services, isolate the responsible code, and then test fixes. With the rise of coding agents, this chain becomes a bottleneck. OpenAI therefore transformed part of this work into an automated workflow, where agents can take charge of the investigation, propose fixes, and verify recovery.
The first version relied on manual investigations, then on specialized tools. OpenAI’s performance team developed more than 30. Codex then served as an analysis interface: instead of manually searching through code and telemetry, engineers could ask the agent why a service had become slower. One weak point remained, very human after all: the agent diagnosed, but the engineer still had to manage each step.
Une chaîne de tâches plutôt qu’un simple chatbot
To reduce this dependency, OpenAI designed a Codex skill named perf-forensics, which encodes the team’s investigation procedures. Once tested, it allowed Codex to receive a performance alert and deliver a diagnosis in a single run. The process soon revealed another limit: engineers acted as intermediaries between monitoring alerts, Codex, and Slack. So why not connect alerts directly to Codex sessions in the cloud?
This is where Performance Factory takes its current form. The system organizes work into tasks that progress step by step: alert, investigation, cause identification, code modification, review, verification. Each task keeps its state, notes, and structured metadata, such as when the regression started. Agents don’t dump mountains of raw logs but useful metrics, queries to rerun, and main conclusions. The file remains usable without turning into a digital attic.
La revue adversariale, ou l’art de contredire l’agent
The first concrete application of this system is the Regression Workflow. When a configured alert triggers, a task is created and the agent starts its investigation using perf-forensics. It first spots what changed, then looks for the cause from available artifacts: code changes, logs, traces. But OpenAI observed that agents sometimes stopped at the first plausible explanation, even when the evidence was insufficient.
To correct this bias, the company added an adversarial review step. A second Codex session receives the complete file with fresh context and clear instructions: look for holes in the reasoning, verify evidence, test other hypotheses. In one case cited by OpenAI, an agent had attributed faulty ChatGPT conversations to image download errors; the reviewer noticed that the internal service responsible for locating files was already restored, while image requests continued to fail. Diagnosis rejected, then. Not glamorous, but effective.
According to the performance team, this review initially sent many cases back for further investigation. Instructions were adjusted, and the return rate is now about 25% of diagnoses. Once the diagnosis is validated, Codex proposes a fix and opens a pull request; an engineer then decides on the merge. Of more than 600 investigations that led to a root cause, about 400 resulted in a proposed fix. Among these, around fifty fixes actually deployed were influenced or cross-checked by agent suggestions, and about fifteen exactly matched pull requests written by Codex.
Ce que Performance Factory dit des limites de l’IA en production
These figures mainly show the gap between diagnosing a problem and delivering a reliable fix. OpenAI acknowledges that its results remain below its ambitions for automated remediation. Agents work better when the cause is clear and the change contained; a serialization duplicate, for example, is fairly easy to spot. As soon as a feature adds latency and requires architectural modification, proposals become more fragile, sometimes too complex, sometimes outright counterproductive. A cache can speed up everything but introduce stale data. Classic.
The company therefore extended the adversarial review to the fixes themselves. After deployment, another agent checks whether the performance issue has disappeared. If not, the task returns to investigation. OpenAI also revised the presentation of results for engineers: grouping related incidents, prioritizing by estimated impact, short reports with charts and metrics. The goal is clear: prevent human teams from drowning in noise generated by agents, even as they still must decide, arbitrate, and validate.
Vers une optimisation plus proactive
OpenAI says it wants to go beyond just reacting to configured alerts. The team is exploring automatic comparisons between CPU profiles, memory data, and request traces across multiple versions to detect slowdowns invisible to classic monitoring. It is also working on rightsizing workflows, that is, adjusting the size of resources allocated to a service to avoid waste or undercapacity.
Ultimately, the company is even considering allowing some low-risk fixes to be deployed without human validation. For now, the line remains cautious. But Performance Factory already offers a glimpse of what agentic engineering workflows could become: less repetitive debugging, more organizational memory, and procedures teams can reuse from one service to another. In production systems, technical knowledge is quickly forgotten; here, it is recorded, replayed, and passed on, which changes the game for teams tasked with maintaining speed under pressure.
Key points
- More than 1,700 investigations in just over three months.
- Over 600 root causes identified by OpenAI.
- About 25% of diagnoses returned for adversarial review.
- About 15 fixes were written exactly by Codex.
- OpenAI is already using Performance Factory for ChatGPT.
By the numbers
- 1,700+ investigations — in just over three months, according to OpenAI.
- 600+ root causes — identified by the internal system, according to OpenAI.
- 30+ tools — developed by OpenAI’s performance team.
- 25% — share of diagnoses returned by adversarial review.
- 15 fixes — exact pull requests written by Codex and shipped, according to OpenAI.