Cloudflare and OpenAI pilot network signals to expand ChatGPT Search

Expanding the search index

Cloudflare and OpenAI recently shared results from a research pilot designed to improve how ChatGPT Search discovers web content. By leveraging a small portion of Cloudflare's network, the companies aimed to identify pages that are often missed by traditional search methods. As reported by shattered.io, the initiative successfully helped the search crawler find and index hundreds of millions of previously unknown pages.

The pilot began in July 2026 and focused on a cohort representing about 1% of Cloudflare's total network reach. While 1% sounds small, Cloudflare's infrastructure reaches within 50ms of 95% of the world's internet-connected population. This means even a small slice of the network provides a massive surface area for discovering new data across the global web.

The primary goal for OpenAI was to determine if network-level insights could help its crawler prioritize pages worth indexing. For Cloudflare, the objective was to prove that traffic data can serve this purpose without requiring the company to hand over actual page content. The companies emphasized that the pilot used web metadata for discovery rather than sharing page content for model training.

How the discovery signals work

Cloudflare sits in front of a large portion of the internet, giving it a unique vantage point to see which pages receive real traffic. The company can identify which pages have changed recently, which URLs lead to dead ends, and which addresses duplicate existing content. These signals are then distilled and passed to OpenAI's crawler to guide its efforts.

The signals run in two directions to optimize the crawling process. Some signals point the crawler toward high-quality pages that deserve to be fetched, such as a restaurant's menu or a specific user guide. Other signals tell the crawler where to avoid wasting requests, such as sites that redirect almost every address to a homepage or pages that return the same bytes as previous visits.

The companies are also testing page-level safety signals to improve the quality of the search index. For example, they are working on a way to flag a single fake login form planted on a legitimate site for exclusion. This allows the crawler to ignore the malicious element while still indexing the rest of the domain, improving the safety of the final search results.

Limits on discovery and ranking

It is important to note that a discovery signal does not grant the crawler automatic access to a website. Domain owners still maintain full control over their content, meaning a page that blocks OpenAI's crawler via robots.txt stays blocked even if Cloudflare flags it as fresh. Cloudflare explicitly stated that these signals are meant to assist the crawler, not bypass existing owner permissions.

Furthermore, these signals do not determine the final ranking of the pages within the search results. OpenAI evaluates and ranks the discovered pages independently, so being included in the index does not guarantee a specific position or a citation in an answer. Publishers should be careful not to assume that being discovered by the pilot automatically leads to a massive increase in referral traffic.

The pilot's results are currently focused on reach rather than specific business outcomes like click-through rates. While the companies are now looking for ways to measure how discovery translates into value, the current data primarily proves the scale of the reach. For site owners, the first visible effect might simply be an increase in crawler traffic in their server logs as new pages are evaluated.

The broader Cloudflare strategy

This pilot is part of a larger move by Cloudflare to position itself as a key intermediary between web crawlers and site owners. In the same three-week period, Cloudflare opened a Web Search API in beta and introduced continuous behavior checks for Verified Bots. These moves show a strategy of selling visibility to one side of the web while providing more control and security to the other.

The Verified Bots program requires crawlers to identify themselves, respect robots.txt, and back off when they receive an HTTP 429 response, which tells a visitor to slow down. Cloudflare measures how consistently a bot follows these rules to create a trust band. This system aims to catch bots that exhibit abusive behavior or fail to honor site preferences, regardless of their identity.

Cloudflare has already demonstrated its willingness to enforce these standards by de-listing certain bots in the past. By integrating these behavior checks with the search discovery signals, Cloudflare is building an accountability loop. This ensures that as AI search matures, the crawlers using Cloudflare's data are held to a standard of respectful and predictable behavior.