Major AI firms propose shared signal for website owners to opt out of training

A unified signal for publishers

Google and Microsoft have joined OpenAI and Anthropic in supporting a shared signal for website owners. This proposal allows publishers to tell AI companies whether they can use website content to train their models. Currently, website operators often have to create unique rules for every different company's crawler to achieve this.

The new initiative aims to simplify the process by letting participating providers interpret a single entry in the same way. This would reduce the administrative burden on site owners who want to manage their data permissions more efficiently. As reported by Mezha.net, the move is intended to provide a clearer path for content creators to set boundaries.

The proposal is based on an Internet-Draft published by Anthropic's Kevin Kelley. It is not a formal standard approved by the IETF, which is the organization that manages internet protocols. Because it is an individual draft, it has no formal standing and is set to lapse on May 3, 2027, unless it is revised.

How the robots.txt signal works

The mechanism works by adding a specific policy token to the User-Agent line of a robots.txt file. This file is a standard way for websites to tell automated programs which parts of a site they can or cannot visit. The new token is specifically labeled as '__usage_ai-training'.

It is important to note that this signal does not block bots from crawling websites. Instead, it acts as a preference for content collection. A crawler collecting content solely for training would skip pages it cannot use, while still allowing the bot to fetch the pages for other purposes.

The IETF's AI Preferences working group defines training as changing a generative model’s learned parameters. This definition excludes models used only for tasks like ranking or classification. OpenAI has stated it will honor the signal specifically for foundational generative AI training, which may be a narrower scope than the draft's definition.

Varied implementation by tech giants

While the four companies agree on the signal, they plan to connect it to their products in different ways. Google intends to treat a shared-token opt-out as a refusal of its Google-Extended instruction. This setting also governs grounding in Gemini Apps, meaning an opt-out could impact citations and clicks in Gemini answers.

Microsoft is taking a different approach by adding a separate, page-level AI Grounding preference. Their DisallowExtended option allows content to remain available for products like Copilot while excluding it from the grounding services they sell to other businesses. This gives businesses more granular control over how their data is used across different services.

The timeline for adoption varies among the major players. Anthropic is currently the only one of the four companies that has started honoring the signal. OpenAI plans to implement it over the next few months, while Microsoft and Google are both aiming for 2027 to begin compliance.

Limitations and remaining hurdles

The proposal is a request rather than a technical block. Because it relies on crawlers choosing to comply, it does not provide a hard barrier for data collection. Furthermore, the draft allows for private agreements to override the public signal. This means some companies might still find ways to access data through other legal or technical means.

Publishers who want to stop all crawlers entirely will still need to use the per-bot rules they use today. OpenAI, Google, and Anthropic have all confirmed that their existing controls for blocking crawlers will remain in place. The new signal is strictly about how content is used for training, not about whether a bot can visit a page.

Meta has not committed to honoring the signal, though its VP of communications called it an important step. The company has faced previous backlash over its crawling terms and has recently removed training from some listed uses. However, some media groups argue that without an express ban, training remains a possibility for some platforms.

The broader landscape of AI data

This move comes as the industry faces increasing pressure regarding data licensing and media rights. News Corp has recently sued Brave Software over the alleged unauthorized scraping of thousands of articles to train AI models. These types of legal battles could eventually redefine how media companies are compensated for their content.

The Trump administration has also shown support for OpenAI in its copyright lawsuit with The New York Times. The administration argues that using data to train AI models may qualify as fair use. This legal environment creates a complex backdrop for the voluntary signals being proposed by the tech giants.

Additionally, the industry is grappling with security issues, such as reports of Iran-linked operators using models to fabricate identities. While these issues are separate from training signals, they highlight the ongoing challenges of managing large-scale AI systems. The proposed robots.txt signal is one small step in a much larger effort to balance AI development with publisher rights.

What to watch next

The primary thing to watch is whether the signal becomes a formal IETF standard or remains an individual draft. If it remains a draft, its long-term adoption by smaller AI companies will be uncertain. It is also worth watching if Google and Microsoft maintain their different approaches to grounding and citations.

Another key factor is how publishers react to the lack of a payment mechanism. Many media organizations have sought direct payments for training data, and this signal provides a way to say 'no' without a way to say 'yes, for a fee.' The evolution of these commercial terms will likely dictate how much high-quality data remains available for training.