Joint research pilot uses network signals to expand AI search coverage, improve crawling efficiency and identify potentially harmful content as AI-driven discovery reshapes how people access information online.
Cloudflare and OpenAI have reported early progress in a joint research initiative that has enabled OpenAI to discover and index hundreds of millions of previously unknown web pages. The collaboration aims to help ChatGPT surface more relevant, timely information while improving how AI search systems discover, evaluate and access content across the internet.
Launched in July 2026, the research pilot uses signals from Cloudflare’s network to help OpenAI’s search crawler identify potentially useful webpages. The pilot has involved a participating cohort representing approximately 1% of Cloudflare’s total network.
The initiative addresses an emerging challenge for website owners and publishers: creating valuable content is not enough if AI systems cannot discover it. Installation guides, product experiences, technical documentation and frequently updated service information can be useful to people even when these pages receive few links from other websites.
As users increasingly turn to AI assistants to research products, solve problems and make decisions, discoverability is evolving beyond traditional search rankings. AI systems must identify relevant sources, assess their usefulness and connect users with original information through answers, citations and website visits.
“For those who choose to make their content available to AI systems, these improvements make discovery easier and more efficient.”
Expanding the reach of useful content
Cloudflare provides network signals that help OpenAI discover more potentially relevant webpages. OpenAI independently evaluates and ranks those pages according to their relevance and usefulness, determining which information should be surfaced to ChatGPT users.
The companies believe this approach can expand the range of sources available to AI-powered search, helping users find specific information that might otherwise remain difficult to locate.
The initiative also complements OpenAI’s broader efforts to improve ChatGPT as a destination for learning, shopping and finding answers through more interactive and tailored experiences. Better web discovery can help these experiences draw on useful, up-to-date information rather than relying on a limited pool of easily discoverable pages.
However, identifying a URL is only the beginning. A page may no longer exist, redirect visitors to another destination or simply duplicate information available elsewhere. Automated traffic can also generate repeated requests for nonexistent pages, consuming resources without adding value.
Improving crawling efficiency and security
The pilot is testing signals that help identify these patterns before crawlers fetch pages. Filtering out nonexistent or redundant destinations can reduce unnecessary requests, while change signals can help determine when content merits another visit.
This approach is intended to balance the initial increase in crawling required to discover new pages against the longer-term need to avoid repeatedly downloading unchanged information.
For website operators, more efficient crawling could reduce unnecessary infrastructure load. For OpenAI, it could allow search resources to be directed towards content that offers greater value to users.
Security is another area under evaluation. The partners are testing signals to identify potentially harmful webpages, including individual pages compromised by attackers to host fraudulent login forms.
Identifying suspicious pages individually could help protect users without automatically penalising an entire domain when only one page has been compromised. Early findings suggest that these signals can help identify pages that should be excluded from search indexes.
Website owners retain control
As AI-powered discovery becomes more important for publishers and businesses, the companies say participation and access decisions remain with website owners.
Cloudflare emphasises that a signal identifying a webpage does not grant permission to access it. Website operators can use its AI Crawl Control tool to understand and manage how AI systems access their websites.
The research uses web metadata for search and discovery rather than sharing page content for model training, according to the companies.
The next phase will focus on improving page selection, testing faster and more reliable signals for detecting content changes, and measuring what happens after pages are discovered. Cloudflare and OpenAI also plan to examine whether improved discovery translates into meaningful business outcomes for publishers and website operators.
The results point to a broader shift in online visibility. As people increasingly use AI to find information, businesses may need to consider not only how their websites perform in conventional search engines, but also whether AI systems can discover and accurately reference their content.
For publishers, the opportunity lies in reaching audiences through new discovery channels. For website owners, the challenge will be balancing that opportunity with control over access, security and infrastructure costs. Better discovery could help close the gap between valuable information being available online and that information actually reaching the people who need it.