Google shows an AI agent that patches its own outage, under four security checks

A outage built for panic

As developers.googleblog.com reported, Google Cloud researchers built a reference design called Project Chrysalis around one familiar emergency: a flash-sale traffic surge. A cache (a fast short-term store, here Google Cloud Memorystore) drops its cold entries, and every request falls through to the database behind it. The database's pool of connections fills up, queries time out and checkouts start failing.

The post highlights a trap in this scenario. Under stress, both tired humans and naive AI assistants tend to restart the database connection pool. In a live stampede, the post says, that makes things worse: it locks the database and pushes delays up almost tenfold.

The demo's baseline shows the damage. After a small warm-up, a burst of 50 requests hits 15 cold inventory keys. The cache hit ratio falls from 100% to 0.0%, the database pool saturates (2 of 2 connections locked) and HTTP 500 errors reach 72.7% to 80.0%, which is 40 failed checkouts.

The agent writes its own fix

The agent in this design is built with Google's Agent Development Kit and powered by Gemini. When it diagnoses pool exhaustion and finds it has no traffic-shaping tool, it writes one. The tool is a token-bucket rate limiter, which hands out a limited number of request "tokens" and refills them at a steady pace. This one has a 10-token capacity and refills at 40.0 requests per second.

The agent mounts the limiter into the API gateway's middleware, the layer that handles requests before they reach the application. In a second 50-request surge, errors fell to 0.0% and database pool saturation fell from 100% to 0%. The slowest-case delay (P99 latency) recovered to 18.2 ms. Forty excess requests were turned away with an HTTP 429 ("too many requests") response, and 10 were served from the warm cache.

Across the simulated outages, the post reports time to first mitigation dropping from 15 to 45 minutes to 1.8 seconds. These are simulation figures from a demo application, not measurements from a customer's production system.

Learning from what worked

After the incident, the system reviews what each action did. It uses a method called Shapley attribution, which splits credit for an outcome among the steps that led to it. Here it compares latency, error rates and database saturation before and after each step. The database restart is penalized because latency spiked to 3,800 ms, while the rate limiter earns +0.97 credit.

The lesson is then stored as a signed rule in a vector search index, a memory that finds entries by meaning rather than exact wording. The rule says that during a cache stampede a database restart is the wrong move and load shedding is the right one. Future alerts can check it.

The agent also proposes a bigger change: a permanent Traffic Shaping Specialist Agent added beside its diagnostic agent. Because this alters the agent's own structure, execution pauses until the on-call engineer approves with a cryptographic token. The change is made in memory, so, per the post, there is no downtime and no container restart.

Four gates before anything runs

The design's main idea is that the agent may adapt its tactics, but its authority to act stays tightly bounded. The post describes four checks that every patch, memory and structural change must clear:

• Sanitization: Model Armor screens incoming alerts for prompt injection (hidden instructions planted in data) and redacts sensitive information. A code-tree check blocks forbidden system calls in anything the agent writes.
• Provenance: Cloud KMS signs each artifact, and Binary Authorization refuses to mount anything whose signature does not verify.
• Policy and human sign-off: Open Policy Agent enforces rules such as a minimum refill rate of 10.0 requests per second, so the agent cannot throttle its own service into an outage. Higher-risk changes need an engineer's approval.
• Audit trail: Each sanitization, check, signature, verdict and approval is written to Google Cloud Logging.

Testing the result before trusting it

The last phase replays incident traffic in BigQuery against 400 alternative rate-limiter settings. The post says the sweep took about 0.5 milliseconds and found a best-performing policy: 5-token capacity, a 5.0 requests-per-second rate and zero server errors.

A red-team agent then attacks the evolved system in an isolated sandbox. It purges 100% of the cache and fires 200 concurrent requests. The new specialist agent absorbed the surge with 0 HTTP 500 errors and a maximum database load of 13%.

Treat this as a blueprint rather than a product. It shows how the pieces fit together, and the strongest idea is the ordering: let the agent propose, verify the proposal, and keep a person in charge of anything structural.