Google, Microsoft, OpenAI and Anthropic agree on shared AI training opt-out

A unified standard for web content

Google, Microsoft, OpenAI, and Anthropic have announced a joint effort to simplify how website owners opt out of generative AI training. Until now, publishers often had to identify individual crawlers and maintain separate rules for every different AI provider. This new agreement allows a single note on a website to block training across all participating companies simultaneously.

The move is designed to reduce the complexity of navigating different platform preferences for the same use of content. According to the Google Search Central Blog, website owners will no longer need to manage separate instructions for every AI developer. This shared signal aims to create a more streamlined experience for creators who want to protect their intellectual property.

While the companies agree on the principle, they have not all committed to the same timeline. Microsoft announced it will begin honoring the new preferences on February 1, 2027, while Google intends to support the signal in early 2027. OpenAI and Anthropic have not provided specific dates, though Anthropic has already begun building support for the specification.

How the policy token works

The new mechanism uses a specific 'policy token' within the existing robots.txt file, which is a standard file used to tell web crawlers which pages they can visit. Instead of naming a specific company crawler, site owners can use the token __usage_ai-training to express a preference. The draft specifies that the double underscore prefix makes it unlikely for the token to be matched accidentally by other systems.

The technical design of this signal is intended to be compatible with existing rules. It introduces no new grammar or directive names to the Robots Exclusion Protocol, which is the standard for robots.txt files. This means that parsers that do not recognize the new token will behave exactly as they did before, ensuring that the change does not break existing web configurations.

The draft also clarifies that wildcard rules, such as User-Agent: *, will not be interpreted as matching the new policy token. If a site owner does not explicitly use the __usage_ai-training token, the system assumes no preference has been expressed. This ensures that current robots.txt files remain unchanged in their current behavior until the owner chooses to add the new signal.

Impact on search and grounding

One of the most important distinctions in this agreement is how it affects search results and 'grounding.' Grounding is the practice of tying a generated AI answer to specific retrieved web pages for better factuality. The companies have clarified that the new training opt-out signal does not affect a site's ranking in search results or its ability to be indexed by search engines.

However, the impact on grounding varies between the providers. Microsoft stated that the new signal refuses training but leaves grounding untouched. In contrast, Google's plan suggests that using the shared token to opt out of training may also withdraw content from grounding in Gemini Apps and Vertex AI, unless a specific allow rule is provided.

This creates a slight difference in how publishers might manage their content. A site owner using Google's system might have to choose between a broad training opt-out that also removes them from Gemini answers, or a more specific configuration. Google notes that publishers can still tailor instructions to individual developers using existing tools like Google-Extended.

Definitions and limitations

The agreement relies on a specific definition of what constitutes AI training. It is defined as using an asset to modify the learned parameters of a generative AI model. This means that models used for non-generative tasks, such as classification or ranking, are generally excluded from this specific training category. Retrieval systems that provide answers at the time of a prompt also fall outside this scope because they do not change the model's parameters.

The draft also establishes that private agreements can override these public preferences. If a publisher has a specific content deal with an AI company, that private contract will govern their relationship regardless of the robots.txt signal. Additionally, the preferences apply to the content at the exact moment it was retrieved by the crawler, meaning it does not retroactively apply to data gathered before the token was added.

Finally, the signal is a statement of preference rather than a strict command for crawling behavior. While the draft suggests that crawlers may refrain from fetching resources that are opted out of training, it does not strictly forbid the crawl itself. A site owner who wants to block a bot from visiting a page entirely must still use the standard crawler name in their robots.txt file to ensure the bot stays away.