An AI pipeline that triages and fixes its own software bugs

Automated triage and repair

A developer recently tested a custom AI pipeline designed to catch, explain, and repair software bugs automatically. During a test on a staging site, a bug was intentionally planted in a database read that caused a search box to crash. The system detected the error and produced a fix request in forty-two seconds after the first failure.

The entire process from the initial error to a ready-to-review pull request took 31 minutes. This pull request included a one-line fix and a test to prove the correction worked. Throughout this process, a human only had to make two specific decisions: approving the fix specification and signing off on the final result.

The developer built this system to maintain full control over how the AI behaves. Because AI can hallucinate or lose track of context, the goal was to create a pipeline where every agent step is checked by something other than itself. This ensures that while the AI does the work, the human remains the final gatekeeper for quality and intent.

How the pipeline works

The system relies on two small services to manage the workflow. The first service, called Night Watch, monitors the site's logs to identify incidents. It then asks an AI to triage the problem and sends alerts to various communication platforms like Telegram and Slack.

The second service, Day Shift, takes over once an incident is identified. It writes a technical specification for the fix, builds a workspace to run the code, and executes the project's checks. It even employs a second agent to review the changes before deploying them to the staging environment.

The workflow is written in plain code to allow the pipeline to stop and resume without repeating work. To ensure reliability, the developer uses 'drills' to break the staging site on purpose. These drills test the logs, the alerts, and the human response, essentially acting as unit tests for the entire maintenance pipeline.

Lessons from the drill

The bug drill revealed several important technical hurdles during the testing phase. For instance, the system initially struggled to confirm if a bug was actually healed because it only looked for the absence of errors. The developer learned that the system must instead replay the specific request that failed to confirm the fix.

Another issue occurred when the spec writer read the main branch instead of the branch containing the bug. This caused the AI to guess the cause of the problem rather than identifying it precisely. While a human note eventually fixed the issue, the goal is to improve the tool so it does not require manual intervention for such tasks.

The developer also noted that alerts need strict budgets to prevent notification fatigue. During one test, a single bad start triggered sixteen emails in a short period. The developer plans to separate the budgets for drills and real incidents to ensure that important alerts are not drowned out by testing noise.

Performance and costs

The financial and time costs of the automated pipeline appear to be low for the developer. The specific bug drill cost $2.45 in agent credits and was completed in 31 minutes. If the two human wait points were removed, the developer estimates the process would take only about 12 minutes to complete.

To test the system's limits, the developer loaded the staging environment with 10,000 realistic accounts. This resulted in 2.57 million entries and 66,881 invoices, totaling 1.74 GB of data. Despite this load, most pages stayed under 0.2 s of data time for the heaviest accounts.

One specific owner-only report was initially slow, taking 2.9 s to load. After the system applied two small indexes, that time dropped to 0.27 s. These results suggest the pipeline can handle significant data loads while maintaining high performance for the end user.

The roadmap ahead

The next phase of development focuses on moving from luck to consistent metrics. The developer plans to implement 'honest healing,' where the system replays failing requests to confirm fixes. They also intend to introduce synthetic users that perform short journeys on the site every few minutes to catch bugs without needing real traffic.

A 'bug zoo' is also in the works, consisting of a set of planted bugs with pre-written answer keys. This will allow the developer to grade the system nightly on how well it triages, specifies, and fixes different types of errors. The goal is to produce a single honest number, such as the median time and cost to heal a bug.

Once these numbers are established, the developer intends to open-source both Day Shift and Night Watch. The ultimate goal is to move toward full automation where the system can resolve issues entirely on its own. The developer describes the current progress as starting small with the intent to eventually go big.