Cloudflare announced improvements to the Cloudflare AI Search user experience to simplify the creation of a search engine for files and websites owned by users, enabling AI agents to access up-to-date data and provide better answers. Previously, the process required assembling several components from the Cloudflare platform, such as Workers AI, AI Gateway, Vectorize, R2, and Browser Run, while AI Search now automatically connects these components.
The service is available for use today and targets developers who want to make structured and unstructured data available to their agents, ranging from individual files to websites. For now, the data source must be a domain that exists in a Cloudflare account, although the company said it plans to offer additional ownership-verification methods later.
Website Indexing and Public Endpoints
The updates include the ability to add websites that do not have a sitemap through the “Discover” analysis option, which discovers pages by following links. Users can also index a collection of data sources within a single namespace and then obtain two public endpoints, /search and /mcp, for searching across multiple instances or websites without authentication.
Cloudflare also allows users to place a custom domain in front of the public endpoints, such as search.example.com/mcp, to customize the service’s address and branding. When the search needs to be private, Cloudflare Access can be added in front of the domain, requiring users to sign in and limiting use to authorized people or agents.
The company also provides an AI Search add-on for EmDash websites, an open-source content management system, to add semantic search to website content. AI Search relies on hybrid search, which combines semantic search with keyword search, making it suitable both for open-ended questions and for finding specific names or terms.
Using the Service in Cloudflare Dev Stack MCP
Cloudflare demonstrated how to use AI Search in the Cloudflare Dev Stack MCP server, which is available for testing through AI Playground. The server combines up-to-date documentation with cited references from the Cloudflare developer ecosystem, helping coding agents rely on newer information instead of outdated training data.
The company created a separate instance for each of its surfaces, including documentation, the blog, API documentation, the community, Astro, Vite, Vitest, Hono, Replicate, and OpenNext, then combined them into a single search through a shared namespace. This can be implemented through a Worker when integrating search into an existing application or MCP server, or by using the public endpoints directly without writing code or deploying an additional service.
The server can also be connected to coding agents through an MCP URL to obtain current, source-backed answers about building applications on the Cloudflare platform. Cloudflare already uses AI Search in its blog search, developer documentation, and Cloudflare.com, in addition to integrating it with the EmDash-built blog site.
Experimental Pricing
AI Search remains free during the beta testing phase, and billing has not yet been activated. Cloudflare says it will notify customers before billing begins, while the prices currently displayed remain subject to change before general availability.
- Basic indexing consumption: $0.75 per million tokens, with a free allowance of 5 million tokens per month across all Workers plans.
- Image processing: an additional cost of $0.50 per million tokens.
- Stored data: $2 per gigabyte per month, with 10 gigabytes free.
- Semantic queries, including hybrid and vector search: $0.75 per 1,000 queries, with 2,000 free queries per month.
- Full-text search: $0.10 per 1,000 queries.
Embedding and reranking operations are free when using the default models or selected models from the Workers AI catalog, while third-party models are billed separately. Cloudflare explains that answer generation and query rewriting are optional steps and are charged according to the Workers AI model selected by the user, or users can use AI Gateway credits with any model or provider.
AI Search follows robots policies. It uses Browser Run for crawling, identifies itself with an independent robot identity named Cloudflare-AI-Search, and follows robots.txt and websites’ robot controls.