Major AI labs back new standard to block training data scraping

The problem with current scraping

Website operators currently use a file called robots.txt to manage how automated programs, or crawlers, interact with their content. This file acts as a set of instructions that tells search engines which pages they are allowed to visit and which ones they should ignore. For years, this has been the primary way to manage web traffic and indexing.

However, the rise of generative AI has made this system much more complicated for publishers. To prevent an AI company from using their data for training, a website owner often has to create unique entries for every single AI provider. If a company uses multiple different scrapers to gather data, the owner must write a separate instruction for every single one of those tools.

As a result, managing these permissions has become a tedious and manual task for content creators. This friction has led to ongoing debates about how to balance the needs of AI development with the rights of the people who create the underlying data. The new standard aims to remove this technical hurdle by creating a universal signal.

How the new standard works

The proposed standard would allow website operators to block all scrapers that collect training data for generative AI models using just one entry in the robots.txt file. This change is designed to be broad enough to cover the entire industry while remaining specific enough to be useful. It targets the specific act of using data to train models rather than just general web crawling.

Crucially, the new setting would not apply to models that perform exclusively non-generative tasks. This means that search engines or other tools that do not use the data to train a generative model would still be able to access the content. The goal is to create a clear distinction between using data to learn and using data to provide a service.

The structure of these files follows a formal framework known as the Robots Exclusion Protocol. The major tech companies involved have already submitted a draft of their proposal to the Internet Engineering Task Force. This nonprofit organization is responsible for maintaining the official standards for how robots interact with the internet.

Key players and timelines

Google, Microsoft, Anthropic, and OpenAI have all announced their plans to support this new crawling standard. These companies represent some of the largest players in the AI space, making their collective agreement significant for the future of web scraping. By agreeing on a shared standard, they are attempting to create a predictable environment for website owners.

The timeline for implementation varies across the different companies. Microsoft and OpenAI both intend to add support for the new standard within the coming months. Google has indicated that it will follow suit sometime next year, providing a longer window for the industry to adjust to the new rules.

Meta Platforms Inc. has also weighed in on the situation. While Meta expressed support for the core idea behind the new technology, the company has not yet committed to actually implementing it. This leaves Meta as a notable outlier among the major tech giants currently involved in the discussion.

Special features for AI grounding

Microsoft is planning to go a step further by providing a specific setting called AI Grounding. This feature is intended for situations where an AI model needs to access a webpage to answer a specific user question in real time. It is different from training data collection because it involves the model seeking out information to provide a current response.

The AI Grounding setting will allow website operators to choose whether they want to allow these types of requests. This gives publishers more granular control over how their content is used in live interactions. It ensures that a site can be