Follow the latest coverage, related explainers and connected technology stories.
Cloudflare is introducing a system that measures whether verified bots respect site preferences, such as robots.txt and HTTP 429 responses, rather than merely verifying their identity. The company will turn these observations into a trust range that could eventually be used to make decisions about allowing automated visits.
Cloudflare launched a new setting called Disallow AI Training that allows website owners to prevent their content from being used to train artificial intelligence models while keeping it available to search crawlers operated by organizations that comply with transparency controls. The initiative includes Apple, Google, and Microsoft, with certain crawlers and control over AI summaries remaining subject to separate restrictions.