Cloudflare announced on September 15, 2026, a new setting called Disallow AI Training that allows website owners to reject the use of their content for training artificial intelligence models without blocking search crawlers capable of keeping the site discoverable. The setting targets the problem of mixed crawlers that use the same crawler for search and model training, which had effectively forced site owners to accept both uses or block both simultaneously.
The setting is available to all Cloudflare customers, on all plans, and can be configured at the domain level from the security settings. The company says that Apple, Google, and Microsoft comply or have committed to complying with this option under a framework it calls Accountable. This classification is granted to crawler operators that provide mechanisms to refuse AI training and AI summaries, transparency about the pages used, and assurances that refusal will not affect traditional search results.
How does the new setting work?
Bot Preference Sync publishes an appropriate preference in the robots.txt file. When Disallow AI Training is selected, mixed crawlers classified as Accountable remain permitted to access the site for search purposes, while other training crawlers—including crawlers operated by Amazon, Anthropic, Meta, and OpenAI—are blocked without affecting the site's appearance in search through these organizations.
Cloudflare also redefined some of its existing options. The Block and Block on pages with ads options now apply to mixed crawlers such as Applebot, Bingbot, and Googlebot, meaning that using them may block both search and training. The names Block AI Bots and Managed Robots.txt will also be retired in favor of separate controls for search, training, and agents, with existing customer settings migrated to the new system.
What differs among the companies?
Apple allows website owners to refuse training through the Applebot-Extended robots.txt rule. It also currently supports AI-summary preferences through the nosnippet directive, but it does not yet provide a tool for inspecting pages at the URL level. Apple stated that refusing training does not affect search rankings.
Google supports the Google-Extended rule and provides a switch in its webmaster portal to exclude content from generative search results, along with reports on search results and AI summaries. Microsoft currently enables control through the NOARCHIVE tag and tools for blocking URLs or removing content in Bing Webmaster Tools, but it plans to add support for a no-training preference in robots.txt at the domain or site level in early 2027. Until then, the new Cloudflare setting does not automatically transfer this preference to Bing.
What changes in practice for website owners?
Most existing customers will not need to take immediate action, as Cloudflare will migrate current settings while preserving their practical effect. However, anyone who wants to block Applebot, Bingbot, or Googlebot completely—including access for search purposes—will need to select Block instead of Disallow AI Training. For new domains, Cloudflare will suggest preset settings that vary according to the site's reliance on advertising, because blocking traffic or replacing it with a generated answer may affect the monetization model.
The editorial reading from certi.news
The actual change here is not merely the addition of a rule to robots.txt, but the shift from a binary option to a more detailed policy that connects the crawler's identity with the purpose of the visit. This is important for publishers who want to preserve search visibility without granting general consent to model training, but it does not eliminate the need to verify that each operator is actually complying with the published preferences.
AI summaries remain a different issue from training: a summary may reduce visits to the site, but it may refer fewer users with higher purchase intent. Therefore, by early next year, Cloudflare plans to provide more precise control over the amount of content that appears in summaries through a central setting, instead of managing the preference separately with each operator. The interoperability of standards, including ai-prefs, and the behavior of agents used on behalf of humans also remain open questions.