SIAgentNews.com
← All news
DevelopingClaudeAgents

How the Claude Code team automates software verification

By building robust verification loops, developers can delegate more work to AI agents while maintaining high code quality.

· First reported by claude.dev

Blueprint drawing: picture for the story 'How the Claude Code team automates software verification'
AI-made blueprint drawing for this story. No real people or products are shown.
90%of code written and reviewed by Claude

The short version

  • Claude Code writes and reviews approximately 90% of its own code.
  • Verification loops allow developers to run hundreds of agents in parallel instead of just one or two.
  • End-to-end tests are prioritized because they are difficult for agents to fool and provide definitive done criteria.
  • Property tests use libraries to generate thousands of inputs to find edge cases that humans and agents might miss.
  • Differential tests compare old and new systems with the same inputs to identify breaking changes during migrations.

Why it matters

This shift moves the human role from reading every line of code to defining what 'done' looks like and reviewing evidence. It enables a high-velocity development cycle where AI handles the heavy lifting of testing and iteration.

Scaling with verification loops

On the Claude Code team, the vast majority of the software—roughly 90%—is written and reviewed by Claude. To manage this scale, the team relies on verification loops as the fundamental building block for delegating work to AI agents. Without these loops, humans would have to manually check every output, limiting the system to running only one or two agents at a time.

By providing agents with a way to verify their own output, the team can run dozens or hundreds of agents in parallel. This system improves quality by allowing the agent to request and run its own tests, benchmarks, and specifications. Once these tests are generated, they run as software, making them cheap to execute frequently without consuming extra tokens.

The team also uses these results to close the loop and improve performance on specific tasks. For instance, if an agent repeatedly fails a test regarding user data logs, the team can create a specific rule in a configuration file to prevent future errors. This process ensures that once a mistake is identified, it is systematically blocked from recurring in the codebase.

Three core testing strategies

The Claude Code team identifies three specific types of tests that are most worth investing in for AI agents. If a developer must choose only one, they recommend focusing on end-to-end (E2E) tests. These tests describe real user flows, such as adding an item to a cart or checking out, which are difficult for an agent to fool because the entire feature either works or it does not.

Property tests are used to find the edge cases that people often miss, which is where agents typically struggle. In this method, a developer writes a statement they believe cannot fail, and a test library generates thousands of different inputs to try and prove it wrong. These are particularly useful for rewrites where the original specifications might have drifted from the actual behavior of the production code.

Differential tests are highly effective for large-scale migrations where the goal is to keep functionality identical. These tests run both the old and new systems with the same inputs to see if the results differ. To make this work, the team uses randomly generated inputs or samples of production traffic while normalizing irrelevant factors like whitespace to ensure the comparison is accurate.

Simulating the human experience

The team argues that running an app end-to-end matters more than running isolated unit tests. They believe that because Claude can test an app the way a human or QA engineer would, it should be given 'hands and eyes' on the real system. This includes using tools like simulators, API harnesses, or computer use capabilities to boot, drive, and inspect the application.

To make this consistent, the team codifies how to start a simulator or run an app into a specific 'verify' skill within the repository. Claude is instructed to use this skill before committing any changes, ensuring it performs a 'smoke test' on its own work. This allows the agent to iterate and catch real mistakes faster, mimicking the way a human developer would troubleshoot a new feature.

When a check fails, Claude attempts to fix the issue automatically. If the failure persists after three rounds of attempts, the system stops and brings in a human reviewer. The people who own the code write these checks and refine them until they flag the same issues a human owner would. This ensures that humans only intervene when the automated verification reaches its limit.

Governance and review rules

To keep the verification process honest, the team maintains a reliability checklist with 39 items. Each item has a specific trigger that Claude reads from the code changes to determine if a check is necessary. This prevents reviewers from having to memorize every past outage, as Claude only flags the 3 to 10 items relevant to the current diff.

The team also uses a REVIEW.md file to guide human reviewers on what to prioritize for specific code sections. They ensure that the rules governing these reviews are kept on a separate branch so that the agent cannot modify the rules it is being judged against. Additionally, they reserve reviews for specific commits, as a passing review on an out-of-date commit provides no useful information.

The team also tracks the performance of the reviewers themselves by monitoring how many merged changes are later reverted. They use a formal evaluation where they recreate the repository to a specific state and provide a one-sentence answer key for bugs. This helps them tune the system to catch important bugs while ignoring 'nits' or pre-existing issues that the author is not responsible for fixing.

Continuous improvement loops

The Claude Code team constantly refines its verification rules by extracting patterns from human comments and PR reverts. If a common issue appears in at least two different pull requests, it is turned into a potential rule. Three agents then try to disprove the rule, and a human finally reviews and approves it for the codebase.

They also replay 'bug finders' against pull requests associated with known past incidents. If at least two finders catch the bug, it is considered a detectable gap that needs a new rule in the review documentation. This proactive approach ensures that the system learns from every mistake made in the past, preventing the same errors from reaching production again.

Finally, the team treats the verification layer as a system that requires regular maintenance. They prune old rules and tidy up test suites to ensure they stay relevant and efficient. By keeping the verification layer out of reach of the changes it judges, the team ensures that the AI remains honest and that the human role shifts toward defining the criteria for success.

What to watch

  • How the 'observe' and 'enforce' modes of commit gates will affect developer velocity.
  • The effectiveness of using three agents to 'disprove' new rules before they are finalized.

Try it

Create a 'verify' skill in your repository that codifies how to run your app in a simulator or local environment, and instruct your AI agent to run it before every commit.

Sources and links

First reported by claude.dev. Written by the SI Agent News Desk from the reporting above.Written with AI from the source above. Every number is checked against the source. Spot a mistake? Corrections.

Liked it? Say why

Add a line for other readers.

No account needed. We never show your email. Comments with links, ads or abuse are held and not shown. Your comment shows after a quick check.

More AI news