Major AI firms agree on shared robots.txt line to opt out of training

A unified standard for creators

Major technology companies including Google, Microsoft, OpenAI, and Anthropic have announced plans to support a shared robots.txt line. This initiative is designed to allow publishers to block their content from being used to train generative AI models with a single rule. By coordinating on this standard, the companies aim to provide a more streamlined way for creators to manage their data across different platforms.

The proposal originated as an Internet Draft submitted by Anthropic engineer Kevin Kelley to the AI Preferences working group at the IETF. The draft suggests a specific policy token named __usage_ai-training that can be placed under a User-Agent line in a website's robots.txt file. This file is a standard text file that tells web crawlers which parts of a site they are allowed to visit or index.

Currently, different AI companies often have their own unique methods for opting out of training. This new shared signal would mean a publisher could write one line of code to communicate their preference to multiple providers at once. While the draft is an individual Internet-Draft and not an official IETF standard, the backing of these major firms suggests a significant shift in industry cooperation.

How the new signal works

The proposed mechanism works by using a specific group of rules in the robots.txt file. For example, a site could write a rule that disallows the __usage_ai-training token while allowing access to other areas like a press kit. The draft specifically defines AI training as the act of using content to modify the learned parameters of a generative AI model. It clarifies that models performing non-generative tasks, such as ranking or classification, are not included in this definition.

A key technical detail in the draft is that a general instruction like User-Agent: * is prohibited from matching the new token. This ensures that the opt-out remains specific and cannot be accidentally overridden by broad instructions. Furthermore, the draft specifies that a file without the __usage_ai-training group indicates no preference, which is the current state for most published files.

It is important to note that this token is framed as a way to communicate a preference rather than a command that alters what a crawler actually retrieves. It does not control the basic crawling of a website. Instead, it tells the AI provider how they may use the data they have already crawled. This distinction helps separate the act of discovery from the act of model training.

Different roles for Google and Microsoft

While the companies are cooperating on the shared line, they have different plans for how they will interpret it. Google stated that when a site opts out using the shared line without specifying their specific Google-Extended tag, they will interpret it as if the publisher had disallowed Google-Extended. This existing tag controls whether content can be used to train future Gemini models or for grounding in Gemini Apps and on Vertex AI.

Microsoft is taking a slightly different approach by introducing a distinct AI Grounding choice that applies only to individual pages. They announced they will begin honoring the new options on February 1, 2027, and will stop recognizing older tags as training or grounding options after March 1, 2027. This move will also replace the NoArchive and NoCache tags that Microsoft has used since 2023.

Microsoft also clarified that using these restrictions will limit the visibility of a brand in answers generated by Artificial Intelligence. However, they noted that these restrictions will have no effect on a site's ability to be listed in search results from Bing. This distinction ensures that publishers can opt out of training without necessarily disappearing from search engine results.

Timelines and industry support

The timeline for implementation varies across the major players involved in the announcement. Anthropic reported that they have already started building support for the draft. OpenAI stated that they aim to implement the signal within the coming months, which suggests they may be the quickest to adopt the new standard.

Google plans to start supporting the control in early 2027, while Microsoft has set its firm date for February 2027. These dates provide a clear window for publishers to prepare their technical configurations. Because none of the three companies currently support the line, the transition period will be essential for testing and verification.

The broader industry is watching these developments closely to see if other major players will join the coalition. Meta's communications head, Andy Stone, voiced support for creating a common signal for multiple providers. However, he did not explicitly state whether Meta will honor the specific robots.txt token proposed in the current draft.