Major AI firms agree on shared opt-out token for web crawlers

A unified signal for publishers

Four of the largest developers in the artificial intelligence space have reached an agreement to honor a shared policy token. This mechanism allows website operators to tell multiple AI companies at once that their content should not be used for training models. The move is intended to simplify the process for publishers who currently face a growing list of different companies to contact individually.

The proposal was originally put forward by Kevin Kelley, a technical lead for standards at Anthropic. It is currently being processed through the Internet Engineering Task Force, which is the organization responsible for developing the technical conventions that make the web function. By adopting this shared token, the companies hope to create a more streamlined way for the internet to manage data permissions.

As reported by BigGo Finance, the agreement involves OpenAI, Google, Microsoft, and Anthropic. These four companies have committed to recognizing a single signal that collapses separate declarations into one line of configuration. This shift is a response to the increasing volume of automated traffic from crawlers that harvest articles, images, and other materials to build larger models.

How the robots.txt token works

The proposed token will reside inside a website's robots.txt file. This is a plain-text document that has been used for decades to tell automated crawlers which parts of a site they are allowed to index or archive. By placing the token here, a site owner can broadcast a clear instruction to any crawler that reads the file.

Under the current system, a publisher who wants to keep their material out of AI training must communicate that choice to every developer separately. This patchwork becomes increasingly difficult to manage as the number of model builders continues to multiply across the industry. The shared token removes this burden by providing a universal way to communicate preferences.

Search visibility and grounding

Microsoft noted that a website's choice to opt out of AI training will not impact its appearance in Bing search results. This distinction is important for publishers who want to maintain their reach in traditional search engines while still restricting how their data is used to build new models.

The company also plans to offer more granular settings for these preferences. These options will allow publishers to choose whether they want to withhold content from AI training or from AI grounding, which is the method of pointing a model toward a trusted source to limit the creation of false information.

Industry reactions and alternatives

Meta has signaled interest in the new standard even though it has not yet made a formal commitment to honor the token. Andy Stone, the company's communications head, said creating a common signal is an important step. He noted that the company looks forward to working with other AI firms on the standards and their implementation across the industry.

Other technical and commercial countermeasures already exist for publishers who want to control their data. For example, the Really Simple Licensing standard allows websites to charge crawlers for access to their content. Additionally, Cloudflare has started blocking AI crawlers by default for its customers to provide an immediate layer of protection.

The timeline for adoption

The speed at which this new signal becomes a standard will depend on how quickly it is adopted by companies outside of the initial four backers. Because Google and Microsoft do not plan to support the token until next year, the mechanism will not be universally observed in the near term. This creates a staggered rollout for the new technical convention.

Publishers who are currently weighing whether to deploy the token may find that the calculus differs from one crawler to the next. Since the arrangement is voluntary, the signal only works if the specific crawlers choose to respect it. This lack of an enforceable rule means that some companies may still harvest data even if a site carries the token.