Google, Microsoft and OpenAI to support shared AI training opt-out

A unified signal for AI training

Google, Microsoft, and OpenAI have announced plans to recognize a shared robots.txt line to manage AI training preferences. This initiative aims to let website owners communicate a single preference to every AI provider that participates in the system. Currently, publishers often have to write specific rules for each individual company's crawler to keep their content out of training sets.

The proposal originated from an Internet Draft submitted by Kevin Kelley to the AI Preferences working group at the IETF. This draft introduces a specific policy token called __usage_ai-training. This token allows a site to specify whether it allows or disallows the use of its content to modify a generative AI model’s learned parameters.

The draft clarifies that non-generative tasks, such as ranking or classification, are exempt from this definition. It also notes that a file without this specific group indicates no preference, which is the current state for almost all published robots.txt files. This shared line does not control crawling, only the specific use of content for training purposes.

How the different giants will respond

While the goal is a shared standard, the three companies will handle the signal in distinct ways. Google plans to treat the shared opt-out line as a block for its Google-Extended tool. This control determines whether content can be used to train future Gemini models or be used for grounding in Gemini Apps and on Vertex AI.

Microsoft is taking a more granular approach by splitting training and grounding into separate preferences. They are introducing a new option called DisallowExtended, which allows grounding in tools like Copilot but prevents it through Microsoft-provided services used by other businesses. Microsoft also allows meta tags or HTTP headers to override the robots.txt file for specific pages.

OpenAI has expressed its intent to honor the signal within the next few months. However, their current crawler documentation does not yet specify what happens if a website provides conflicting rules between the new shared line and the existing GPTBot rule. Anthropic has also begun development to support the feature, though they noted it will take some time to implement across all services.

Impact on existing web settings

For many website owners, these changes will require a transition from older tags to new standards. Microsoft specifically announced that it will stop recognizing the older NoArchive and NoCache tags as options for training or grounding after March 1, 2027. They are replacing these with the new preferences to provide clearer control over how content informs AI-generated answers.

Google's implementation is tied to its existing Google-Extended system. If a site uses the shared line to opt out of training but does not specify Google-Extended, Google will interpret that as a disallowance of Google-Extended. This means the shared line will effectively block both training and grounding for Google's Gemini ecosystem simultaneously.

Sites that currently use NoArchive or NoCache to limit how Copilot uses their content must take specific actions to stay protected. Microsoft will open a temporary preference recording in Bing Webmaster Tools on December 15 for sites still using those old tags. Pages without a new preference by the 2027 deadline will default to allowing grounding until the next crawl.

The roadmap to 2027

The timeline for these changes varies across the major providers. Google aims to support the shared control by early 2027, while Microsoft will begin honoring the new options as they crawl starting February 1, 2027. The draft itself is currently set to expire on May 3, 2027, unless a new version is proposed to replace it.

Kevin Kelley, who submitted the draft, intends to propose his design as a change to a separate Content-Usage draft. This suggests that the vocabulary for how AI interacts with web content is still being refined by the technical community. For now, the new shared line will not be considered for any rules that are already written into existing robots.txt files.

The move represents a step toward a more organized way for publishers to manage their digital assets. By creating a common language for AI preferences, the industry hopes to reduce the complexity of protecting content. However, because different companies have different architectures for training and grounding, the practical effects of the shared line will vary depending on the platform.

What this means for publishers

Publishers who want to maintain control over their content will need to monitor these specific dates and requirements. Because the shared line does not reach the same uses at every company, a single opt-out might not be a universal shield. For instance, a site could potentially block training while still allowing grounding at Microsoft, but it cannot do that at Google because both are tied to the same control.

Website owners should be aware that these changes are primarily about training and grounding, not general search visibility. Google and Microsoft have both stated that these new preferences will have no effect on a site's ability to be listed in standard search results. The primary goal is to give creators a clearer way to say no to the use of their data in model development.

As the 2027 deadlines approach, the technical community will likely see more updates as Google and OpenAI share more details. For now, the most important takeaway is the shift toward a standardized vocabulary. This move signals an effort to move away from a