One team says AI agents should write their own runbooks instead of being retrained

The claim in plain terms

As dev.to reported, an engineering team behind two projects, AriaWave and VisonLabs, argues that fine-tuning is the wrong tool for making AI agents dependable in real operations. Fine-tuning means further training a model so its internal numbers, called weights, shift toward your material. The author says that approach broke down as soon as a cloud provider changed a command-line flag or an API endpoint, because the model kept acting on outdated documentation.

The post opens with a quick history of blame. Teams first pointed at limited context length, then at retrieval-augmented generation (RAG, where a model looks up documents as it works), then spent what the author puts at fifty thousand dollars on custom weights. In the author's telling, none of these stopped the same mistakes from recurring.

What a runbook is and what goes in it

A runbook is a written, step-by-step procedure for a recurring task, the kind a senior engineer keeps for fixing a firewall rule or a broken webhook. The author's point is that people do not rewire their brains after a fix. They write it down.

The team's agents, built on a setup the post calls the Hermes Agent fleet, do the same. After solving a non-trivial infrastructure failure, an agent updates a skill file before closing the task. The post lists four parts of each file:

• Trigger conditions: when to load the runbook, and when not to, so the agent's base instructions stay fast.
• Verified command sequences: the exact commands and settings that actually worked on that machine.
• A pitfalls ledger: every trap found along the way, so the same mistake happens only once.
• Verification gates: the proof the agent must produce, such as an HTTP 200 response, a screenshot or a database row count, before it reports a task done.

The examples the author gives

The post says the fleet now holds 92+ of these files. The examples range widely: recovering a 60GHz rooftop radio over an IPv6 link-local connection, reconciling a ledger across several entities in QuickBooks, and rendering a 4K 60fps cross-dissolve in a video tool called HyperFrames. The listed pitfalls include path-translation quirks on Windows and a browser pop-up that blocks page closing.

The author's line is that a clever prompt gives you a demo, while a library of tested runbooks gives you a team that gets better over time. Their advice is to give the agent a file system, hold it to strict proof-of-success checks, and have it write its own procedures.

Why the idea is appealing, and where it can fail

The practical appeal is simple. A runbook is plain text. If it is wrong, a person can open it, fix it or delete it, and the behavior changes immediately with no training run and no special hardware. The knowledge also lives outside the model, so it can move with you if you switch to a different model.

There are real weak spots, which are our analysis rather than claims from the post. The agent may load the wrong file. Files go stale when the systems they describe change. A large pile of files can crowd the model's working memory, the context window, and cost more to run. Because a self-written instruction is text the model trusts, bad information picked up from a web page and saved into a runbook could stick around. Human review of new runbooks is the sensible guardrail.

Fine-tuning still has jobs runbooks do not do well, such as locking in a consistent output format or squeezing a task onto a smaller, cheaper model.

What the post does not show

Treat this as an engineer's field report, not a study. The post does not present a side-by-side test of runbooks against fine-tuning, error rates or costs. The headline claim that runbooks beat fine-tuning rests on the team's own experience with their own systems. The post also points readers to the authors' own projects and videos, so it is partly promotional.

Even so, the order of operations it suggests is sensible: try the cheap, reversible fix first, and consider retraining only when text files clearly cannot solve the problem. The two approaches are not mutually exclusive, especially with open-weight models you can modify yourself.