OpenAI to honor shared robots.txt signal for AI training preferences

Simplifying web permissions

OpenAI announced on October 30, 2026, that it will begin honoring a shared signal within robots.txt files. These files are the standard tools websites use to tell automated systems how they can interact with their content. By supporting this shared signal, OpenAI intends to help site operators communicate their preferences to many different AI providers at the same time.

Currently, website owners face a significant administrative burden when trying to manage AI access. They must identify individual crawlers from various companies and maintain separate, unique rules for every single provider. This new approach aims to streamline that process by allowing a single preference to apply across all participating providers who recognize the signal.

The company described this move as a step toward an open web where information remains discoverable but owners have clearer ways to manage their choices. OpenAI stated it will share specific implementation details in the coming months as it works to roll out the change. The company also noted that this new signal will not replace the existing controls they already provide through robots.txt.

The technical mechanism

The new signal is defined in an Internet-Draft titled Machine-Readable Training Preference Signal. This draft was authored by Kevin Kelley of Anthropic and is based on the existing robots.txt standard published as RFC 9309. It introduces a specific policy token called __usage_ai-training to be used in the User-Agent line of a robots.txt file.

The design ensures that existing internet parsers will not be confused by the new addition. Because it uses the standard path language and group structure, older parsers will simply ignore the policy tokens they do not recognize. This means the new signal is independent of traffic-management signals, which control how a crawler moves through a site.

For example, a website owner could use the token to disallow AI training for their entire site while still allowing it for a specific folder like a press kit. The draft specifies that wildcard groups, such as User-Agent: *, must not be interpreted as matching the policy token. If no group in a file contains the token, the system assumes no preference has been expressed.

Standardizing AI vocabulary

The signal relies on a specific vocabulary to ensure different companies interpret the rules the same way. This vocabulary defines AI Training as the use of an asset to modify the learned parameters of a generative AI model. It specifically excludes models used for tasks that are exclusively non-generative.

The vocabulary also distinguishes between different categories of AI interaction. It separates AI Training from AI Use, which covers assets used as input to a model rather than provided directly by a user. It also includes a Search category for applications primarily designed to select assets and direct users to their locations.

Under these definitions, each category can be assigned a value of allowed, disallowed, or unknown. If a website owner provides conflicting preferences, the most restrictive one will apply. While the vocabulary does not guarantee that every recipient will follow the rules, it provides a common language for those who choose to do so.

Cooperation and existing tools

OpenAI is joining other participants in the Internet Engineering Task Force (IETF) AI Preferences working group to promote this standard. The group includes a wide range of organizations, including Google, Microsoft, Meta, and News Corp. OpenAI is encouraging other AI providers to honor the signal to create a more consistent experience for web publishers.

This new shared signal works alongside OpenAI's current crawler controls. For instance, GPTBot is the specific agent used to crawl content for training generative AI models. Site owners can still use the existing GPTBot rules to indicate if their content should be excluded from that specific training process.

Other bots like OAI-SearchBot and OAI-AdsBot continue to function independently. A webmaster can choose to allow OAI-SearchBot to appear in search results while simultaneously using the new signal to disallow training. These settings remain independent, allowing for granular control over how different types of AI technologies interact with a website.