SIAgentNews.com
← All news

How to add llms.txt and tdmrep.json files to a Next.js site

A dev.to tutorial shows Next.js site owners how to publish guide files for AI tools, and how to avoid robots.txt mistakes that confuse AI crawlers.

· First reported by dev.to

10 minutessetup time the guide claims for the endpoints

The short version

  • The guide says the setup takes less than 10 minutes on a Next.js App Router site.
  • A valid llms.txt starts with one H1 title, then a short summary in a blockquote.
  • Link text should describe the page; the ## Optional section marks pages a model can skip.
  • Under RFC 9309, Disallow: /admin also blocks /admin-guide.html, because rules match by prefix.
  • Cloudflare's managed AI-bot blocking can add rules above your own robots.txt and cause mixed policies.

Why it matters

If you run a website, AI assistants and AI search tools are now among its readers. These small files are a cheap way to say which pages matter and what may be done with your content. They are requests, not locks, so they work best alongside a clear decision about your own policy.

A to-do list for AI-friendly sites

As dev.to reported in a practical tutorial, people who run developer docs, API platforms or web apps built with Next.js (App Router) can publish a few machine-readable files for AI tools. The guide names coding assistants such as Cursor, Windsurf, GitHub Copilot and Claude Code, plus AI search crawlers such as OAI-SearchBot and PerplexityBot, as the intended audience.

It covers three things: the exact layout of an llms.txt file, three common robots.txt errors that can trip up AI crawlers, and how to serve both llms.txt and a /.well-known/tdmrep.json file from a Next.js project. The author says the whole job takes under 10 minutes.

What goes inside llms.txt

Think of llms.txt as a short, plain-text table of contents written for machines. Unlike a sitemap, which is a bare list of addresses, it is meant to be readable by both strict software parsers and language models. The format follows a specification published at llmstxt.org and is strict about order.

The file opens with a single top-level heading (the project or platform name) on the very first line. Right after it comes a short quoted summary that states plainly what the platform does and who it is for, with no marketing language. After that come sections of links, each with a descriptive label and a one-line explanation.

The guide highlights three rules to check:

  • One H1, on line 1. Parsers expect the heading first, followed immediately by the summary.
  • No vague link text. Labels like "Click here" give a model nothing to judge by. The label and the description after the colon are what it uses to decide whether a page is worth fetching.
  • Use the ## Optional section. Links placed there tell a model with limited working memory which pages it can skip.

Serving the file from Next.js

In the App Router, you create a route handler, a small piece of server code that answers requests for a given address. The guide places one at app/llms.txt/route.ts, marks it as static so it is built once rather than computed on every visit, and returns the file contents as plain text.

The example also sets caching instructions: public, max-age=3600, s-maxage=86400, stale-while-revalidate=604800. In plain terms, browsers and shared caches may keep the file for set periods, and an older copy can still be served while a fresh one is fetched. The response also carries an X-Robots-Tag header set to "all".

The same approach, the guide says, works for the tdmrep.json file at its .well-known address. That file follows a W3C protocol for text and data mining, which lets a site state whether its content may be mined.

Where robots.txt quietly goes wrong

The guide's second half is a warning about robots.txt, the older file that tells crawlers where they may go. Under the IETF standard RFC 9309, rules match by prefix. That means Disallow: /admin blocks /admin, /admin/ and also /admin-guide.html, which you may not have meant to hide.

The opposite slip also happens. If you write Disallow: /private/ with a trailing slash, a request to /private without the slash is still allowed.

The third trap involves Cloudflare. If you use it in front of Next.js, its managed robots.txt feature ("Block AI training bots") may add its own rules above the ones your app generates. The author says this often leaves AI bots such as Amazonbot, Claude-Web and Meta-ExternalAgent facing inconsistent rules.

Useful, but not a lock

The article ends by pointing to a client-side tool, KivaroTools llms.txt & AI Crawler Generator. It is said to convert an existing sitemap or HTML docs into llms.txt, llms-full.txt, tdmrep.json and Next.js route code. It also profiles token usage, a measure of text length for models, across GPT-4o (128k), Claude 3.7 Sonnet (200k) and DeepSeek R1 (64k). Treat that as a convenience the author recommends, not something the guide proves out.

One caution beyond the source: these files are signals, and crawlers are not obliged to follow them. Decide what you want AI tools to read and mine before you publish anything, then check that your files do not contradict each other.

What to watch

  • Whether major AI crawlers actually honor llms.txt and tdmrep.json, since neither is binding.
  • Whether Cloudflare's managed robots.txt rules clash with your own policy after you publish.

Try it

Open your robots.txt and test two paths: a Disallow rule without a trailing slash (does it catch pages you wanted public?) and one with a slash (does the bare path slip through?).

Sources and links

First reported by dev.to. Written by the SI Agent News Desk from the reporting above.Written with AI from the source above. Every number is checked against the source. Spot a mistake? Corrections.

Liked it? Say why

Add a line for other readers.

No account needed. We never show your email. Comments with links, ads or abuse are held and not shown. Your comment shows after a quick check.

More AI news