A free scanner shows big developer sites still hide their APIs from AI agents
A dev.to author scored four homepages on how easily AI agents can read and use them. The basics were covered, but tool discovery and payment files were mostly absent.
The short version
- The scanner checks ten signals and gives a score out of 100; the author built it and the page also sells a setup service.
- On 22 Nov 2026 it scored vercel.com 74, supabase.com 79 and www.cloudflare.com 84; the author's own site got 100.
- All three big platforms served readable HTML, had llms.txt, a sitemap, JSON-LD and explicit AI-crawler rules.
- Only one of the three had an MCP discovery file, and none had an x402 payment catalog.
- The scan cannot say whether any AI assistant will cite or recommend a site.
Why it matters
Websites are starting to have two audiences: people and software agents that read, call and may pay for services. This audit gives you a concrete checklist for the first audience you probably neglect. It also shows that even well-run platforms have not yet made their tools easy for agents to find.
What the author did
As dev.to reported, a developer built a small scanner called Agent-Ready Scan and pointed it at four homepages on 22 November 2026. Three were large developer platforms: vercel.com, supabase.com and www.cloudflare.com. The fourth was the author's own site, manhourplan.com.
The author is open about a conflict of interest. The scanner is their product, and the page that hosts it also offers a paid, done-for-you setup. The scan itself is free, needs no account and does not keep results, so you can check it without handing anything over.
The premise is that AI assistants do not browse the way you do. Many do not run JavaScript, so a page that only fills in after scripts load can look empty to them. They also look for special files made for machines and obey crawler rules written for named bots.
What gets checked
The tool looks at ten signals and adds them into a score out of 100. The heaviest weights go to two things: whether the first HTML response already contains real content, and whether the site has an llms.txt file. That is a short Markdown page at the site root that tells language models what the site is and where its key pages live.
Other checks cover a longer version of that file (llms-full.txt), the site's robots.txt rules for AI crawlers such as GPTBot, ClaudeBot and PerplexityBot, an XML sitemap, and schema.org JSON-LD, which is structured data describing a page's contents.
The rest are about APIs. These include a /.well-known/mcp.json file that points agents to an MCP server (MCP is a standard for plugging outside tools into an AI), and an x402 file listing APIs that charge per call in USDC, a dollar-linked digital currency. They also include an API catalog defined in RFC 9727 and an OpenAPI description, a machine-readable manual for an API.
The author notes that these API-focused checks are irrelevant to a plain brochure site. A company with no API or data to offer can still be in good shape without them.
The scores and what they show
Vercel scored 74 and lacked llms-full.txt, mcp.json and x402.json. Supabase scored 79 and lacked mcp.json and x402.json. Cloudflare scored 84 and lacked x402.json and an API catalog. The author's own site scored 100, which is unsurprising for a site built by the person who wrote the test.
Two patterns stand out. First, the three big platforms had the fundamentals: readable HTML without JavaScript, llms.txt, a sitemap, JSON-LD and deliberate AI-crawler rules. These are no longer a differentiator among major developer brands.
Second, helping an agent discover a tool is still uncommon. Just one of the three exposed an MCP discovery file, and none published an x402 catalog. Those files are how an agent could find a service, and pay for it, without a person reading documentation first.
- Basics: widely adopted by the big three.
- Tool discovery: patchy.
- Pay-per-call listings: absent in this sample.
A checklist you can borrow
The author's advice is practical and ordered by effort. Start with llms.txt, which they call the most valuable file on the list and say takes about ten minutes: a one-line description of your site plus links to your most important pages.
Next, make a deliberate choice about AI crawlers and write it into robots.txt. One common approach is to allow bots that fetch pages for assistants or search (OAI-SearchBot, ChatGPT-User, Claude-User, PerplexityBot) and block those used for training (GPTBot, ClaudeBot, Google-Extended). Allowing everything is also valid. The point is to decide rather than leave it to chance.
Then check your rendering. If your homepage is an empty container until scripts run, most crawlers will see nothing. If you run an API, publish an OpenAPI description and an RFC 9727 catalog that links to it. If you operate an MCP server, add the mcp.json file and list the server in the official MCP Registry.
Limits to keep in mind
Four sites, one of them the author's own, is a small sample. It illustrates a method; it does not describe the industry. Sites also change, so the author warns that a rerun may give different numbers.
The scanner only looks at public files and the raw HTML. It cannot tell you whether any AI assistant will mention or recommend your site, and the author says nobody can honestly promise that. A high score means you are easy to read, not that you will be chosen.
The tool allows 10 scans per IP address per day. The author also offers it as a free MCP tool, so an agent can run the check itself.
What to watch
- Will more large platforms publish MCP discovery and x402 payment files, or will those stay rare?
- Does a higher readiness score translate into agents actually citing or using a site? The scan cannot show this.
Try it
Open yourdomain.com/llms.txt and yourdomain.com/robots.txt in a browser. If the first is missing, write a short description and key links; then decide which AI bots you allow.
Sources and links
- Reportingdev.to: the article we sourced
Liked it? Say why
Add a line for other readers.
No account needed. We never show your email. Comments with links, ads or abuse are held and not shown. Your comment shows after a quick check.
Listen to the AI Wire headlines
Listen to the AI WireListen to our daily AI shows every morning: on your commute, in the shower, wherever you are
Listen to the daily showsFirst reported by dev.to · Read the original. Written by the SI Agent News Desk from the reporting above.Written with AI from the source above. Every number is checked against the source. Spot a mistake? Corrections.
