A developer's API turns web pages into clean Markdown to save AI agents tokens

Why raw web pages are wasteful

When an AI agent reads a web page, it usually receives the page's underlying code (HTML), not the tidy text you see in a browser. That code includes menus, cookie banners, tracking scripts and embedded graphics. The model reads all of it, and it is billed for all of it in tokens, the small chunks of text that models use to measure input and cost.

As dev.to reported, a developer who built a service called WebPulse puts the waste at up to 80% of a model's context window, meaning the amount of text it can consider at once. His example is a typical blog post or documentation page. The raw HTML runs to about 250 KB, or roughly 60,000 tokens, while the readable content is about 15 KB, or roughly 3,500 tokens.

These are the author's own figures for one kind of page, and results will differ from site to site. Still, the direction is plausible to anyone who has looked at a web page's source code.

What the Markdown endpoint returns

An endpoint is a specific web address you send a request to. WebPulse's Markdown endpoint takes a page's URL and sends back a stripped-down version. Markdown is a lightweight text format that keeps headings, lists and links but drops the visual styling.

According to the post, the response includes several things:

• The page as GitHub-flavored Markdown.
• A formatted plain-text version.
• Pre-split pieces of text, called rag_chunks, ready to be stored in a vector database (a database that finds text by meaning rather than exact words). The post names Pinecone, Chroma and Supabase.
• A figure, llm_token_savings_percent, showing how much was saved.

The chunks matter for RAG, short for retrieval-augmented generation. In RAG, an AI looks up relevant documents before answering, and those documents need to be cut into digestible pieces first. The tutorial includes a short Python example that requests the Stripe homepage and prints the savings and a sample of the result.

The bonus tools: SEO and contacts

The same service has two other features. The first is an SEO audit, which checks how well a page is set up to be found by search engines. The author says it runs 25 checks in under 150ms and works directly on the page's code without launching a full browser.

The audit gives a score from 0 to 100 and a grade from A+ to F. It covers the title, description and canonical tag (the tag that tells search engines which version of a page is the official one). It also checks heading order, images missing descriptive alt text, the balance of internal and external links, and basics such as HTTPS. Results come back as lists of critical issues, warnings and passed checks.

The second feature is a contacts finder. It scans a page for corporate and support emails, phone numbers, links to 10 social networks, and key pages such as pricing, contact, about and privacy policy. Anyone using that for outreach should think about privacy rules and the target site's terms first. The post does not discuss either.

How to try it, and what to check

The author offers two routes. A web playground needs no login, so you can paste in a URL and look at the result. Alternatively, the RapidAPI marketplace has a free tier at $0 a month with 50 requests a month, which is enough to test a handful of pages.

Keep some caution in mind. This is a single developer promoting his own product, and we have not run the API. The post gives no independent measurements of output quality, and it does not say how the service copes with pages that load their content through JavaScript. Its speed and savings figures are claims, not test results.

The post also does not say what happens beyond the free tier. If you plan to build on it, check the pricing and limits before you depend on it.