A dev.to guide shows Claude Code running website A/B tests on its own

The slow step after the build

As dev.to reported in a post by a founder of a testing tool, coding agents can now produce a landing page in minutes. The harder question comes afterward: does the new page actually convert better than the old one? Answering that needs a split test, where real visitors see different versions at the same time.

Traditionally that means a testing product, a dashboard and someone to watch it. The post argues the agent can run that loop itself: add the site, install the tracking code, start a test and read the result, all from the terminal.

How the agent reaches the testing tool

The connection runs through the Model Context Protocol, or MCP. It lets an AI app such as Claude Code, Claude, ChatGPT, Cursor or Codex call tools hosted on a remote server. The server publishes a menu of actions, and the agent picks from it.

The post groups the useful actions into four kinds:

• Setup: add a site and check the tracking tag is installed.
• Tests: create, list, read, change or stop a test.
• Evidence: heatmaps, session recordings, lessons from past tests and ideas for what to test next.
• Conversions: record a sale or sign-up from your own backend.

The author uses his own product, Luria, for the demo and says the same idea applies to any testing tool with an API. As an example he points to PostHog, which has an MCP server for feature-flag experiments.

The workflow, step by step

First comes connecting. One command registers the server with Claude Code, and the first call opens a browser where you sign in with an emailed link and approve access. No API key is pasted. Other editors take the same web address in a small settings block.

Next, you ask the agent to add your site and install the tag. It receives a one-line script, edits your layout file, then keeps checking until the tag reports back. Until a site is connected, the other tools reply with the next setup steps, which stops the agent from creating tests on nothing.

Then you ask for a test, for example on the homepage headline, with the current version as the control. The agent can write the new version itself or let the tool draft one from the page and the site's data.

The post stresses counting real outcomes rather than clicks. If purchases happen on your backend, such as in Stripe, the conversion can be sent from there with the visitor's ID, an event name, a value and an order number. You can also ask the agent to wire a Stripe webhook to the tool, since it can see your code.

Finally, you ask how the test is doing. The agent reads visits, conversions and each version's chance of beating the original, then can move on to the next suggested test.

Rules that keep the agent in check

The design includes several safety choices. The first live change waits for the owner's approval: the agent can propose, but a human signs off. After that, tests can run on their own. The original page stays live as a held-out control and never falls below a minimum share of traffic, so results compare people at the same moment, not before and after.

The statistical bar is explicit. A change is kept only when it is 95% likely to beat the original, after at least 200 visits per version. The post also links a sample size calculator for estimating how long that might take.

The author lists guardrails he would want with any tool: human approval of the first change, a kill switch that stops everything fast, spending caps, and read-only tools labeled as such so the app can approve reads automatically and ask before writes. He also wants error messages that say what to send next, so the agent can fix a bad call rather than guess.

Who should care, and the caveats

The idea is more incremental than new. A/B testing is old, and the agent does not change the statistics. What changes is that building, launching and measuring can happen in one working session, with less clicking through dashboards.

There are limits. You need real traffic, and a small site may wait a long time for a trustworthy gap between versions. Someone should still choose the success measure and the stopping rule before launch. Your agent must also support MCP servers, so check its documentation.

Note the disclosure: the author is the founder of Luria and mentions that new accounts get 1,000 free credits. Treat the post as a tutorial from a vendor, and judge the workflow separately from the product. Other pricing is not stated.