SIAgentNews.com
← All news
DevelopingClaudeAgents

Anthropic uses a Claude-driven chaos loop to stress test new Docs launch

Anthropic used an autonomous agent loop to simulate millions of documents, allowing them to cut required server capacity from 500 to 50 before launch.

· First reported by claude.dev

Isometric diorama: picture for the story 'Anthropic uses a Claude-driven chaos loop to stress test new Docs laun'
AI-made isometric diorama for this story. No real people or products are shown.
4.6Mdocuments created in 6 days

The short version

  • Anthropic launched Claude Docs on September 16, 2026, reaching 4.6 million documents by September 22.
  • A Claude agent ran a 'chaos loop' for two weeks, creating over 1 million synthetic documents a day to find system breaking points.
  • The automated testing helped Anthropic reduce the required memory-optimized pods from an initial estimate of 500 down to 50.
  • The process cost approximately $125,000 in inference fees, a task that might have taken a human engineering team about six months.
  • Seven of the ten scenarios rehearsed by the AI in staging actually occurred during the live production launch.

Why it matters

This demonstrates a shift toward 'high-altitude' prompting, where engineers task AI with achieving complex outcomes rather than individual steps. It shows how autonomous agents can handle the tedious work of telemetry analysis and iterative debugging at a scale humans cannot easily match.

Scaling Claude Docs to millions

Anthropic recently released Claude Docs, a collaborative editor where users and the AI can work on the same document simultaneously. The service launched on September 16, 2026, and saw rapid adoption, with users creating over 4.6 million documents in just six days. Despite this massive spike in traffic, the service maintained high stability with only 60 server errors across 2 billion requests.

To prepare for this launch, the team needed to know if their infrastructure could handle millions of documents without crashing. They initially estimated they would need 500 memory-optimized pods to support the expected load. However, through rigorous automated testing, they were able to optimize the system so significantly that they only needed 50 pods to handle the live traffic.

This success was largely attributed to the improved capabilities of frontier models. The team noted that newer models can now maintain a single goal over long periods of time. This consistency allowed them to hit launch scale within a very tight timeline, moving from a small internal proof of concept to a public release in just a few weeks.

The autonomous chaos loop

About two weeks before the public launch, an engineer set up a staging cluster that mirrored the production environment. They tasked a Claude agent with a 'chaos loop' designed to break core components on purpose. The agent was instructed to run a harness that created over 1 million synthetic documents every day to push the system to its limits.

The setup functioned as a coordinator that managed several sub-agents to perform specific tasks. The loop followed a repetitive cycle: the agent would pick a scenario, run the test harness, read the telemetry, propose a fix, and ship that fix to the staging environment. It used specific tools to interact with the system, such as a Datadog MCP to read performance data and a Slack MCP to learn from other infrastructure teams.

The agent was given the freedom to explore many different failure modes, including document creation saturation. It would analyze every database query to spot inefficient patterns and rewrite them. The engineer only intervened for 'hard to undo' decisions, such as changing the database schema or the way documents were stored, which required human approval before being deployed.

Iterative debugging and optimization

One of the most significant wins from the chaos loop involved identifying a major memory leak. Early in the testing, a single large document was consuming about 3 GB of memory on a pod. The Claude agent initially tried to fix this by shrinking a cache, but it quickly realized the memory was actually being held by a separate dependency and dropped that change.

The agent continued to narrow down the problem using finer-grained memory profiling. It eventually discovered that every keystroke was copying the document into multiple different data structures, adding several megabytes each time. Claude then rewrote these structures to use a copy-on-write method, which allowed them to share a single reference instead of holding multiple copies.

After about three days of automated iteration, the memory usage for that large document dropped from 3 GB to around 180 MB. This specific optimization was a primary reason the team was able to cut their hardware requirements so drastically. The agent's ability to perform these many small, iterative steps saved the team from having to manually hunt for the bug.

Lessons in high-altitude prompting

The success of this project relied on a technique described as raising the 'altitude' of a prompt. Instead of asking the AI to fix individual crashes, the engineer asked the model to make the entire service crash-free. This required setting up the environment so the AI could independently pull crash reports and run its own reproduction harnesses.

This approach helps prevent 'goal drift,' where a model might forget the primary objective after many iterations. Because the newer frontier models are built for long-running work, the coordinator agent was able to hold all the project invariants across many cycles. The engineer did not have to constantly restate the goal as the agent moved through hundreds of different changes.

The team also found that using Rust was beneficial for this workflow. While Rust is often difficult for humans to write, the models handle it well, and it provided a clean signal for the AI. By avoiding certain noise, like the unpredictable scheduling of a garbage collector, the models could more accurately trace performance numbers back to specific lines of code.

Cost and engineering impact

The total cost for the inference used during this two-week push was roughly $125,000, based on the list price of the model used. While this is a significant amount, the team noted that it replaced work that would have taken a human engineering team about six months to complete. Most of that human time would have been spent on the tedious tasks of reading telemetry and generating code proposals.

The team believes this method is highly scalable for smaller projects as well. They estimated that a similar push for a smaller service stack, such as a few Kubernetes nodes and a Postgres instance, might cost between $3,000 and $10,000. This suggests that autonomous agents can provide high-value infrastructure hardening even for smaller teams with limited budgets.

The results were validated by the production launch, where seven of the ten scenarios rehearsed in the chaos loop actually occurred in the live environment. Because the agent had already shipped the fixes behind runtime flags in staging, the team was able to handle the pressure by simply flipping a switch. This level of automated preparation provided a level of confidence that would be difficult to achieve through manual testing alone.

}

Sources and links

First reported by claude.dev. Written by the SI Agent News Desk from the reporting above.Written with AI from the source above. Every number is checked against the source. Spot a mistake? Corrections.

Liked it? Say why

Add a line for other readers.

No account needed. We never show your email. Comments with links, ads or abuse are held and not shown. Your comment shows after a quick check.

More AI news