Cloudflare and OpenAI revealed initial results from a joint research experiment aimed at improving how artificial intelligence systems discover useful and constantly changing web pages. The experiment began in July 2026 and relied on signals derived from the Cloudflare network to help OpenAI’s search crawler identify pages worth examining before consuming additional resources to download and analyze them.
The companies say the experiment, which was limited to a participating sample equivalent to approximately 1% of Cloudflare’s total network, enabled OpenAI to discover and index hundreds of millions of new web pages. OpenAI uses these pages as sources that may appear to ChatGPT users, while the final evaluation and ranking of the pages remain within its independent systems based on relevance and usefulness.
From Improving Visibility to Improving Discovery
Cloudflare believes that a site’s appearance in traditional search results is no longer the only measure of its discoverability. When a user relies on an artificial intelligence system to obtain an answer or perform a task, the system must first be able to find the appropriate page and then determine whether it contains original, useful, and accessible information.
Examples include pages that may not receive many links from other sites but remain practically important, such as an installation guide for a specific device, updated passport requirements, or a restaurant’s menu and opening hours. Network signals serve here as a feedback loop that helps OpenAI direct crawling resources toward pages likely to provide actual value.
Reducing Unhelpful Crawling and Improving Safety
The experiment showed that finding a URL does not necessarily mean that valid content exists for indexing. Repeated automated requests for nonexistent files may make a page appear interesting, while some sites may redirect many URLs to the homepage, or multiple URLs may lead to duplicate versions of the content.
Detecting these patterns before fetching a page can reduce unnecessary downloads and the burden on sites. The companies are also testing signals to identify pages that may be harmful, such as a fake login page planted by an attacker inside a legitimate site. Rather than penalizing the entire domain, page-level evaluation makes it possible to isolate suspicious content and keep it out of the index, according to the early results presented by Cloudflare.
What Changes Practically for Site Owners?
Updating every page continuously does not mean that a crawler must redownload the entire site. Signals related to changes can help determine when a page merits a new visit, while filters prevent crawling pages that do not exist or that redirect elsewhere. Discovering new pages may increase crawling initially, but the goal is to keep the index up to date while reducing repeated work on pages that have not changed.
Cloudflare says the next phase will focus on improving page selection, testing faster and more reliable signals for detecting changes, and measuring outcomes after content is discovered. The companies will also continue studying whether improving discovery leads to practical results for site owners, such as increased traffic or access to an interested audience.
Experiment limitations: Cloudflare emphasizes that signaling a page does not grant permission to access it, and that the research uses web metadata for discovery and indexing purposes, not the sharing of page content to train models. Domain owners can use Cloudflare’s AI Crawl Control tool to understand and manage how artificial intelligence systems access their sites.