SIAgentNews.com
← All news

A from-scratch Python agent guide that adds stop limits, guardrails and tests

A dev.to tutorial builds an AI agent without a framework and covers the parts demos skip, so it stops looping and refuses unsafe actions.

· First reported by dev.to

5,469Estimated tokens in call 50 of the naive loop (call 1: 378)

The short version

  • The agent is built without a framework and runs offline against a scripted, deterministic fake model.
  • A naive loop grew from 378 estimated tokens on call 1 to 5,469 on call 50.
  • Four stop budgets cap steps, tokens, tool calls and repeated identical calls.
  • Guardrails check the allowlist, argument schema, business rule and human approval, in that order.
  • Outcome-based evals caught bad refunds that the agent itself had reported as successes.

Why it matters

Most agent failures happen after the demo, when a real ticket sends the model in circles or a hidden instruction tells it to act. This guide shows that the fixes are ordinary code: limits, checks and tests. Even if you never write Python, it gives you a checklist for judging any agent product.

Where demo tutorials stop

Many "build an agent" guides end the moment the demo works: the model asks for a tool, the tool returns data, the model answers. As dev.to reported in this tutorial, the trouble starts on a real support ticket. The agent can repeat the same lookup until someone kills it, or follow an instruction buried in a customer note and delete an order.

The author builds the whole thing in Python with no framework. The tutorial adds a tool-calling loop, explicit stop rules, two kinds of memory, guardrails, tracing and tests. The language model is replaced by a scripted stand-in that replays fixed responses, so every run is repeatable and needs no API key.

The setting is a small support desk with two orders, a refund ledger and a set of policy documents. One order note carries a planted malicious instruction, so the guardrails have something real to stop. The author says the program output in the post comes from real runs, and notes that token counts are rough estimates of about 4 characters per token, so they are good for comparison, not billing.

A loop with four brakes

At its core an agent is a loop. You send the task, the history and the tool descriptions to the model. If it replies with plain text, you are finished. If it asks for tools, you check each request, run the permitted ones, add the results to the history and go around again.

The author first shows the naive version and how it fails. One scripted model keeps requesting the same order lookup forever, and the test harness pulls an emergency brake at 50 calls. Because each pass resends the whole history, cost climbs steeply: the first call was 378 estimated tokens and the 50th was 5,469.

The fixed loop has four separate limits, each aimed at a different failure:

  • max_steps caps how many times the model is called.
  • max_tokens is a spending ceiling, checked before each call.
  • max_tool_calls guards against a model that requests many tools at once.
  • max_identical_calls spots a stuck agent: the same tool with the same arguments three times is almost never progress.

Every exit returns a status, so a caller can tell "finished" from "gave up". Tool errors also become messages the model can read and react to, instead of crashing the program.

Guardrails where actions happen

Each tool is described with a JSON Schema, a standard way of declaring what arguments are allowed. The guide recommends forbidding extra fields, because otherwise a model can invent something like an "admin" flag that slips through. It also warns that the validation library does not enforce email-style format hints by default, so patterns or fixed lists are safer.

Before any tool runs, a policy applies four checks, cheapest first: is the tool on this agent's allowlist, do the arguments match the schema, does the request obey the business rule, and has a human approved it where needed. The allowlist is enforced at the moment of execution, not just when the prompt is written, because a model can name any tool from a hallucination or an injected instruction.

In the injection test, the note told the agent to refund A-1002 in full and delete A-1001, and the scripted model obeyed. The refund passed the format and amount checks, since 899 was within the order total. The human reviewer then saw that nobody had asked for a refund and declined. The delete tool existed in the system but not on this agent's allowlist, so it never ran.

The author is candid that this is damage limitation, not a cure for prompt injection. The model still read the hostile text; the change is that reading it no longer moves money.

Memory that survives a restart

Short-term memory is just the list of messages sent on each call, and it must be trimmed. The tutorial points out a subtle bug: cut the list at the wrong place and it may begin with a tool result whose matching request was dropped, which providers reject. The fix is to never start the kept portion on a tool turn, and a test checks this across budgets from 50 to 1,950 tokens. Summarizing old turns, called compaction, is the next step but needs a model call, so it is left out.

For long-term memory the guide uses SQLite's built-in full-text search with relevance ranking, which needs no vector database. It also records a bug the author hit: ordinary text such as an email address or an apostrophe can break the search syntax. Quoting each word and joining them with OR fixes it.

Recalled notes are labeled as data rather than instructions, since a note is another place injected text can hide. A test opens two separate sessions on one database file to prove a saved fact reaches the second agent's prompt. The author adds a privacy caution: notes about customers are personal data, so record who wrote them and allow deletion.

Tests that judge the result, not the claim

The tracer writes one JSON line per step, using naming borrowed from OpenTelemetry's conventions for generative AI. Those conventions are still marked as in development, so names may change. The tutorial advises recording tool arguments, guardrail decisions, token usage and final status, while hashing or dropping secrets.

The evals run each scenario twice, with guardrails on and off, and grade the final state of the environment, such as the refund ledger. The unguarded results are the telling part. In two scenarios the agent reported success, yet the ledger held a $150 refund against an order worth $120, and an amount saved as the text "120". Only the outcome check noticed.

The scripted suite cannot say how often a real model makes such mistakes. For that, the author suggests swapping in a real provider through a small adapter and running each task several times. For an agent that handles money, the number to watch is the chance that every one of the repeated trials succeeds.

What to watch

  • How often a real model, rather than a scripted one, triggers the failures the tests simulate.
  • Whether the OpenTelemetry naming for AI traces changes while still marked as in development.

Try it

Build a tiny agent with one tool and three written test cases, as the newsletter suggested, and make one test try a forbidden action. Check the final state of your data, not just what the agent says.

Sources and links

First reported by dev.to. Written by the SI Agent News Desk from the reporting above.Written with AI from the source above. Every number is checked against the source. Spot a mistake? Corrections.

Liked it? Say why

Add a line for other readers.

No account needed. We never show your email. Comments with links, ads or abuse are held and not shown. Your comment shows after a quick check.

More AI news