Vedha lets an AI agent test your web app and flags failures the screen hides

The problem: a screen can lie

AI agents are good at operating a website: they open pages, fill in forms and press buttons. What they are weak at is judging whether the action really worked. If a checkout page shows a friendly confirmation, an agent will usually take it at face value.

That is a problem when the server behind the page failed. As dev.to reported in a tutorial for a tool called Vedha, a page can display a success message while the underlying request returned a 500 error (the code for a server-side failure) and nothing was saved. A test that only reads the screen would pass anyway.

Vedha is built to close that gap. It is open source, released under the Apache-2.0 license, and sits on top of Playwright, a widely used tool for controlling a browser from code.

How the setup works

Vedha is an MCP server. MCP, the Model Context Protocol, is the standard that lets an AI assistant call outside tools. Because of that, Vedha can plug into Claude Code, Cursor, VS Code and other MCP-compatible clients with a single command or a short config entry.

One setting is required: you must list the domains the agent may visit. The tutorial uses localhost, so the agent stays on your own development server rather than roaming the open web. You may also need to install the Chromium browser for Playwright once.

After that, you ask the agent in plain language to test something, such as placing an order on a local shop. The agent drives the browser through Vedha, one action at a time, and can state what success should look like for each step.

Checking the server, not just the page

The key idea is a flag the agent can set on any step that changes data. When a click is marked as a data-changing action, Vedha does more than read the page. It watches the network traffic and, where possible, reads the data back to confirm the change landed.

On a healthy app, the tutorial shows the order request returning a 201 (a success code), followed by a check of the order list that confirms the new order exists. On a deliberately broken backend, the page still shows its success text, but Vedha marks the step as failed because the request returned a 500. It also tells the agent not to claim success.

Two safeguards go with this. If a submission times out and nobody can tell whether the order went through, Vedha reports an unknown outcome and will not let the agent press the button again until a read-only check clarifies things. That prevents an agent from creating duplicate orders. The agent can also query server state directly through a separate probe tool without leaving the page.

From one-off run to a lasting test

Once the agent has verified a flow, you can ask it to save the run as a Playwright test. Vedha writes an ordinary test file from the steps that were confirmed. Because the server confirmed the order during the exploration, the generated test includes a server-side assertion as well as a page-text check.

The file opens with a note that it was generated and should be reviewed before it is committed, and it lists its assumptions. That review step is worth taking seriously, since the test reflects only what the agent explored.

There is also a guard option for the rest of your suite. If you swap one import line for a Vedha package, any test that passes its own checks while the app returns a 5xx error, or while a write request fails, will fail instead of going green. That applies to tests written by people as well as by agents.

What the evidence shows so far

The author reports a small pilot: one model, tasks kept apart from development, and three attempts per task. With Vedha the agent missed 9 planted bugs, compared with 26 when it used Playwright MCP. It also used about 30% fewer tokens, the units AI models are billed and limited by.

Those figures come from the tool's own author and from a deliberately small test, so they are a promising signal rather than proof. The tutorial itself says some checks, such as confirming a write was stored, only work where the API allows a read-back.

A sensible way to judge it is to run it on a staging copy of your own app, not your live site, and treat its findings as leads to verify. The project also offers a demo you can run without your own app.