Follow the latest coverage, related explainers and connected technology stories.
Cloudflare is introducing a system that measures whether verified bots respect site preferences, such as robots.txt and HTTP 429 responses, rather than merely verifying their identity. The company will turn these observations into a trust range that could eventually be used to make decisions about allowing automated visits.
Cloudflare has added a dedicated space for bot operators to BotBase, allowing them to track, edit, and cancel submission requests, along with a form describing a bot’s behavior and use of content. The company has also rebuilt the review process to include automated checks that could speed up request approvals.
Cloudflare views AI agents as a new type of web visitor, and says that treating them like traditional bots could mean blocking customers instead of blocking harmful traffic. The company proposes building an open agentic internet based on tools and standards that make content readable, discoverable, actionable, and payable.