With the increasing reliance on Personal AI Agents to perform tasks such as comparing products, searching for support information, or even completing purchases on behalf of users, a new technical challenge has emerged for website owners. These agents act as intermediaries between humans and the internet, causing their traffic to appear in systems as automated traffic, which makes it difficult to distinguish whether this activity serves the business or poses a security risk.
To address this challenge, Cloudflare has launched a set of new analytical tools and granular access policies that allow organizations to welcome these agents on their own terms, rather than resorting to blanket blocking that could lead to lost customers.
Deeper Visibility into Agent Traffic
The new features rely on Cloudflare's massive database of verified bots, adding a new category in the Security Analytics section called Personal Assistants. This allows security teams to monitor this type of traffic separately from other automated bots or humans.
- Precise Statistics: The new Top statistics tool provides a list of agents visiting the site and the number of requests each one sends.
- Path Analysis: Administrators can filter data to see the paths requested by the agent, the status codes received, and the networks from which they originate.
- Account Protection: For sites using the Account Abuse Protection feature, anti-fraud teams can now identify the user represented by each agent, helping to distinguish between normal customer behavior and hacking attempts using AI agents.
Flexible and Customized Access Policies
Cloudflare did not stop at providing visibility; it also introduced greater control by renaming settings to AI Crawler and Agent access policies. These policies allow website owners to choose different levels of control for each supported agent:
- Allow: To ensure the agent has access without restrictions.
- Block on pages with ads: A strategic option that allows the agent to browse content but prevents it from accessing pages that rely on ads for revenue.
- Block: To completely prevent the agent from accessing the site.
Why This Development Represents a Shift in Website Management
This announcement represents a transition from the phase of "fighting bots" to the phase of "managing agents." Previously, dealing with automated traffic was done through blocking or allowing, but with the emergence of agents representing real customers, it has become necessary to understand the context of the request.
The challenge Cloudflare is trying to address is maintaining the relationship between the site and the user when the agent acts as an intermediary. The site may recognize the agent, but it may lack vital information such as: Is this user a subscriber? What permissions have they granted to the agent? Cloudflare aims in the future to find ways to preserve user privacy while transferring user context through the agent, ensuring the continued delivery of the correct service to the real customer via their intelligent intermediary.