Follow the latest coverage, related explainers and connected technology stories.
Cloudflare is introducing a system that measures whether verified bots respect site preferences, such as robots.txt and HTTP 429 responses, rather than merely verifying their identity. The company will turn these observations into a trust range that could eventually be used to make decisions about allowing automated visits.
Meta plans to implement an update to its commercial terms on October 30 that would allow it to use content from websites linked to business accounts for access, indexing, analysis, and training, without a clear opt-out option. While the company says the change is limited to consolidating provisions and does not alter how it crawls the web, legal experts warn that it could expand its authority over publishers and web content.
Cloudflare launched a new setting called Disallow AI Training that allows website owners to prevent their content from being used to train artificial intelligence models while keeping it available to search crawlers operated by organizations that comply with transparency controls. The initiative includes Apple, Google, and Microsoft, with certain crawlers and control over AI summaries remaining subject to separate restrictions.
Cloudflare announced Bot Preference Sync, a feature that automatically updates robots.txt according to site settings for search, agent, and training bots. The feature will be available to all customers, from the Free plan to Enterprise, while retaining the option to disable synchronization.