OpenAI, Google and Microsoft back one robots.txt signal for opting out of AI training
Site owners may soon tell several AI companies "don't train on this" with one robots.txt entry instead of writing separate rules for each crawler.
The short version
- OpenAI, Google and Microsoft each announced they will support a common AI-training opt-out signal in robots.txt.
- The signal is a policy token named __usage_ai-training; Disallow means no training, Allow means training is fine.
- It does not block crawling itself, so existing crawler-specific rules still apply.
- The proposal is a personal-capacity draft filed with the IETF and has no formal standards status yet.
- Each company's timeline differs: OpenAI says within months, while Google and Microsoft point to later dates.
Why it matters
If you publish writing, images or code, you could soon express one training preference for several AI companies at once. It will not enforce itself, but it gives site owners a clearer, simpler way to say no, and gives regulators and publishers a shared reference point.
One signal instead of many rules
As itmedia.co.jp reported, OpenAI, Google and Microsoft have each said they will support a shared way for website owners to state their wishes on AI training. Anthropic is also working on an implementation.
Until now, owners had to identify each AI company's crawler (an automated program that reads web pages) and write a separate rule for each one. The new approach puts a common signal in robots.txt, the plain-text file that tells automated visitors what they may do on a site.
How the token works
The draft, called Machine-Readable Training Preference Signal, adds a policy token named __usage_ai-training to the User-Agent line. Under it, Disallow means "do not train on this" and Allow means training is acceptable.
For example, disallowing everything but allowing a press-kit folder tells AI companies to train on nothing except that folder. If a site does not use the token, it is treated as having expressed no preference.
The signal is separate from crawling rules. It does not stop a bot from visiting; it only states how the content may be used. It also adds no new syntax, so existing robots.txt readers should keep working.
Where each company stands
OpenAI aims to support the signal within the next few months and urged other AI firms to respect it. Google is targeting a later date and plans to treat a site that uses the shared opt-out, without explicitly setting its older Google-Extended control, as opting out of that too.
Microsoft will begin reading the new settings on a set date and adds page-level controls through meta tags and HTTP headers, which will take priority over robots.txt. Anthropic's standards lead, Kevin Kelly, wrote the draft and told a mailing list that Anthropic has begun implementing it, though wider rollout will take time.
The limits
The draft was filed in Kelly's personal name, so it is not yet an official internet standard. Companies may also let their own controls override the shared signal or honor separate contracts differently.
The definition covers training of generative AI models, not models used for other purposes. As with all of robots.txt, it works only if companies choose to follow it.
What to watch
- Will the draft become an official IETF standard, and will more AI companies commit?
- How will companies handle cases where their own controls conflict with the shared signal?
Try it
Open your site's robots.txt file at yourdomain.com/robots.txt and note which AI crawlers you already block, so you can add the shared token once your providers confirm support.
Sources and links
Liked it? Say why
Add a line for other readers.
No account needed. We never show your email. Comments with links, ads or abuse are held and not shown. Your comment shows after a quick check.
Listen to the AI Wire headlines
Listen to the AI WireListen to our daily AI shows every morning: on your commute, in the shower, wherever you are
Listen to the daily showsFirst reported by itmedia.co.jp · Read the original. Written by the SI Agent News Desk from the reporting above.Written with AI from the source above. Every number is checked against the source. Spot a mistake? Corrections.