SI Agent News
DevelopingClaudeOpenAIAgents

A small JavaScript tool checks whether your AI prompts still work after changes

A dev.to tutorial builds a no-API-key test runner that checks saved AI answers against fixed rules, so quiet breakage shows up before users see it.

The short version

Why it matters

If you run a chatbot, an email drafter or any workflow built on a language model, a prompt tweak or a provider update can quietly break something that worked. A regression test turns that worry into a pass/fail check you can rerun after every change. The habit is useful even if you never write the code.

The problem: AI answers drift

Ordinary software tests compare an output with one expected result. Language-model answers don't behave that way. A reply can be worded differently and still be fine, or sound convincing while breaking a rule your product depends on.

As dev.to reported in this tutorial, applications built on language models can regress, meaning something that used to work stops working, when prompts, models or APIs change. The tutorial's answer is a small JavaScript runner that checks answers against explicit, repeatable rules rather than matching one exact paragraph.

What the runner actually checks

Each test case has a stable ID, a short description and a set of checks. The sample cases cover four kinds of rule:

The evaluator is a pure function. It takes a case and some response text and returns pass or fail with a list of reasons. It never calls a model or touches files, which makes failures easy to diagnose.

How it runs without an API

The tool works on saved responses stored in a fixtures file, so it makes no network requests and needs no API keys. That lets you check the checking logic itself before you add cost, credentials or provider quirks.

A command-line script pairs each case with its saved answer, prints PASS or FAIL, and writes a timestamped JSON report. It exits with a failure status if any check fails. A missing saved answer also counts as a failure, since silently skipping a case would hide a problem.

The tutorial also adds tests for the evaluator, using Node's built-in test runner. They cover a passing response, a response with both a missing phrase and a forbidden one, and bad or wrongly typed JSON. The reports leave out response text by default, to avoid copying sensitive output into build files.

What the author refuses to claim

The tutorial is open about its limits. It does not include a live connection to OpenAI or Anthropic, because the sources it relied on did not verify current SDK syntax, model names or request formats. It tells readers to check each provider's official documentation before building that piece, and to keep it separate from the evaluator.

It also warns that a passing run proves little on its own. It shows that these saved examples satisfy these rules. It does not show that a model will always answer correctly, that the rules are complete, or that a provider hasn't changed its behavior.

Phrase checks are described as brittle: equivalent wording can fail, and a phrase being present doesn't make the whole answer right. The author also notes a test-oracle problem. A test that passes today may still encode the wrong expected behavior, so cases should be treated as reviewed product requirements, not as output from an automated system.

Who should care, and what to do

Builders get a clean starting skeleton. Non-engineers get the idea behind it. Our newsletter hosts pointed out that the habit transfers without any code: keep ten to twenty real examples with answers you would accept, and rerun them after every change, including when the provider updates its model.

The tutorial is a foundation rather than a finished product. The rules are simple, the data is saved, and the live-model step is left to you. That modesty is a reasonable trade: you get something you can trust and extend instead of integration code that might not match any provider's current rules.

What to watch

Try it

List ten to twenty real questions your AI workflow handles, write the answer you'd accept for each, and rerun them after every prompt or model change.

Sources and links

Liked it? Say why

Add a line for other readers.

No account needed. We never show your email. Comments with links, ads or abuse are held and not shown. Your comment shows after a quick check.

Listen to the AI Wire headlines

Listen to the AI Wire

Listen to our daily AI shows every morning: on your commute, in the shower, wherever you are

Listen to the daily shows

First reported by dev.to · Read the original. Written by the SI Agent News Desk from the reporting above.Written with AI from the source above. Every number is checked against the source. Spot a mistake? Corrections.

More AI news