Cloudflare says the internet no longer serves a single human audience. Whereas websites once relied on visitors who read content, viewed ads, and paid subscriptions, software acting on users’ behalf has become a second audience that is growing rapidly. According to the company’s figures, the average number of HTTP requests handled by its network rose from 63 million requests per second at the end of 2024 to approximately 115 million currently, with peaks exceeding 150 million requests, while daily requests originating from AI agents grew by more than 1,700% over the course of a year.
Cloudflare adds that more than half of internet traffic was non-human for the first time this year. However, the company distinguishes between three different types of automated traffic: search crawlers, model-training crawlers, and agents that retrieve information or carry out an action at a person’s request. This distinction matters because blocking an agent that books a table or compares insurance offers could mean blocking the human customer behind it, whereas a training crawler may consume content without referring a visitor or generating direct revenue.
From Visits to Paid Use
Cloudflare believes the traditional web model has been disrupted by answer engines that summarize website pages without sending readers to them. As a result, websites bear the costs of bandwidth, computing, and origin-server capacity, while not necessarily receiving an ad impression, subscription, or referral. The company says human traffic declined in some sectors, such as retail, software and IT services, and financial services, by as much as 40% in less than a year.
Cloudflare has therefore introduced two experimental monetization models. The first is Pay Per Use, which allows a publisher to opt in and receive compensation when its content is actually used, rather than being paid merely for being crawled. The company says the buyer determines what constitutes use and the value of that use, while the publisher receives reports on the content used, the time of use, and the revenue, and may also receive data about the questions that surfaced its work.
The Monetization Gateway, meanwhile, targets services, data, APIs, and tools that can be priced per request, query, or token. In the closed test, available to eligible customers in the United States, rules can return an HTTP 402 Payment Required response using the open x402 protocol, allowing the agent to pay the seller directly. Cloudflare says its AI Gateway uses the system to pay inference costs, making the company an early user of the technology.
Agent Visibility and Setting Their Terms
The company offers tools such as AI Crawl Control, Business Insights, and BotBase to show who is crawling a site and which URLs they are requesting. Web Bot Auth also enables agents’ requests to be cryptographically signed; Cloudflare says more than 500 billion authenticated bot requests pass through its network each week. In July, the company replaced a single key for blocking AI bots with separate controls for search, agents, and training, available on all plans, including free ones.
On September 15, Cloudflare launched the Disallow AI Training option to keep a site indexable in search while instructing the operator not to use the data for training. It says Apple, Google, and Microsoft have committed to respecting this option, while Cloudflare Radar publicly tracks crawler behavior. The company also provides Markdown for Agents to reduce unnecessary visual elements for agents, and WebMCP to expose actions directly instead of forcing the agent to guess buttons and interfaces.
Why Does This Development Matter?
The actual change is not merely the increase in the number of bots, but the agent’s transition from a content consumer to a party that can purchase a service or carry out a transaction. This requires website owners to know the identity of the automated visitor, distinguish its purpose, set different terms for search, training, and commercial use, and then measure and financially settle the value.
Nevertheless, Cloudflare presents these models as experimental bets rather than an established standard. The definition of “use” differs between a search engine that cites a source, a research agent that quotes a paragraph, and a shopping agent that completes a purchase. Pricing and discovery have also not yet been settled, and the article does not explain how usage reports will be reviewed or disputes over value and payments resolved. The issue of infrastructure concentration also remains open: Cloudflare advocates standards such as x402 and Web Bot Auth, but it remains a provider of a large portion of the tools for visibility, identity, and collection.
The company says more than 20% of the web is behind its network, and that approximately 80% of leading AI companies use it. It is betting on becoming a shared layer for managing identity, measurement, pricing, and settlement, while leaving the final decision to domain owners. The success of this vision will depend on whether agents and publishers accept these open mechanisms, and on the market’s ability to turn automated visits from an unpaid cost into usage that can be measured and priced.