OpenAI builds autonomous system to find and fix app performance issues
The new Performance Factory uses Codex agents to automatically investigate, diagnose, and propose fixes for software slowdowns.
The short version
- The system tracked over 1,700 investigations and root caused 600 cases in just over three months.
- Codex agents use a 'perf-forensics' skill to access metrics, logs, and traces to identify slowdowns.
- A secondary 'adversarial' agent reviews proposed fixes to check for errors or unintended consequences.
- The workflow automates the entire cycle from initial alert to deployment verification.
Why it matters
This marks a shift from AI as a simple chatbot to AI as an autonomous engineer capable of maintaining complex infrastructure. For users, it means faster updates and more stable performance for products like ChatGPT.
Moving beyond manual debugging
Before the introduction of Codex, OpenAI engineers had to manually sift through massive amounts of data to find out why a service was slowing down. This process involved examining metrics, comparing traces, and debugging live machines, which required deep specialized knowledge of each service. Because signals were easy to miss, the team eventually built over 30 different tools just to make these investigations slightly more manageable.
The team noticed that they were repeatedly facing the same types of problems, such as regressions in the first token latency of ChatGPT. To handle these recurring issues more efficiently, they began using Codex as a partner to help navigate large codebases. While Codex could find relevant metrics and traces, engineers still had to steer the process and decide which specific tools to use at each step.
To move toward full automation, the team captured their manual investigation process into a shared skill called perf-forensics. This skill tells Codex which tools are available and provides a playbook for how to work through a slowdown. By refining this skill with past data, they enabled Codex to provide a diagnosis in a single run without needing a human to guide every individual step of the investigation.
The architecture of the factory
The Performance Factory is an autonomous system designed to run end-to-end agentic workflows in the cloud. It consists of four main components: a task tracker with configurable workflows, a dispatch loop that assigns work to agents, a cloud API for running Codex, and subscriptions to wake agents up. This architecture allows the system to track the state of a task and pass it between different agents or human engineers as it progresses.
Engineers define these workflows by writing configuration files and prompts in Markdown. These files list the various states a task can move through, such as 'needs investigation' or 'fix proposed,' and specify which steps require a human to step in. The dispatch loop periodically checks which tasks are ready and assigns a Codex agent to perform the next action based on the instructions provided in the configuration.
By running these agents in the cloud rather than on local laptops, OpenAI can run work in parallel and handle tasks asynchronously. The system uses subscriptions to listen for events, such as a GitHub review comment or a completed deployment. When an event occurs, the task becomes ready again, and the dispatch loop assigns an agent to pick up where the work left off.
Automating the regression cycle
The first major workflow built in the factory is the Regression Workflow, which handles performance drops from start to finish. When a performance alert fires, the system opens a task that an agent picks up to investigate the slowdown. The agent uses the perf-forensics skill to look back at data, identify which requests were affected, and check if recent deployments or configuration changes caused the issue.
To ensure accuracy, OpenAI implemented an adversarial review process where a separate Codex session tries to disprove the first agent's diagnosis. This reviewer checks for counterfactuals, such as whether a specific rollout actually caused the slowdown or if the issue persisted after a rollback. If the diagnosis is confirmed, the agent then writes a code fix and opens a pull request for an engineer to review.
Once an engineer merges the fix and it is deployed, another agent is triggered to verify the recovery. It checks if the performance has returned to the original baseline. If the recovery is only partial, the agent resolves the original task and opens a new one for the remaining slowdown. If there is no improvement at all, the agent records the result and sends the task back for a new investigation.
Lessons in agentic workflows
Building the Performance Factory revealed that raw AI output can often be a wall of text that is difficult for humans to parse quickly. OpenAI found that they needed to design the system so that engineers could focus on making decisions rather than reconstructing the agent's work. The current workflow now links related incidents and ranks them by estimated impact to help humans prioritize which issues need the most attention.
The team also learned that updating a model or a specific skill can improve some diagnoses while making others worse. Because of this unpredictability, they must test the impact of these updates before deploying them to production. They are currently building an evaluation loop that compares agent-proposed fixes with the final changes merged by human engineers to identify where the AI is falling short.
Looking forward, OpenAI is exploring ways to use synthetic performance tests so agents can measure their own proposed fixes before asking for a human review. They are also investigating how to identify low-risk changes that could eventually be shipped without any human intervention. The goal is to move toward a system where the AI handles the majority of routine maintenance, leaving humans to manage high-level architectural tradeoffs.
What to watch
- Whether OpenAI will allow other companies to use the Performance Factory as a service.
- How many low-risk fixes will eventually be shipped without any human in the loop.
Try it
If you are a developer, look for 'agentic workflows' in your own CI/CD pipeline to see if you can automate repetitive debugging tasks.
Sources and links
- ReportingOpenAI: the article we sourced
First reported by OpenAI. Written by the SI Agent News Desk from the reporting above.Written with AI from the source above. Every number is checked against the source. Spot a mistake? Corrections.
Liked it? Say why
Add a line for other readers.
No account needed. We never show your email. Comments with links, ads or abuse are held and not shown. Your comment shows after a quick check.