Standardizing Digital Content Protection
Google, OpenAI, Anthropic, and Microsoft have announced their commitment to supporting a new directive within the "robots.txt" file, which is the technical standard used by website owners to manage how web crawlers interact with their content. This move aims to provide publishers with a unified and effective way to prevent AI companies from scraping their data and using it to train generative models.
The new proposal, drafted by Kevin Kelley, Technical Standards Lead at Anthropic, is currently under review by the IETF, the organization responsible for establishing open internet standards. This move comes to solve a technical problem facing website owners, who currently must define separate rules for each AI company individually.
How Does the New Standard Work?
Instead of dealing with each company independently, the new standard will allow publishers to use a unified User-Agent specifically for AI training, which appears in the robots.txt file as follows:
- User-Agent: __usage_ai-training
- Disallow: /
This addition allows websites to collectively block crawlers dedicated to collecting training data, with the possibility of exempting certain parts of the site (such as press files) via the "Allow" command. It is important to note that this measure does not prevent search engines from indexing the site or its appearance in search results; rather, its impact is limited to preventing the use of content in model training processes only. For example, Google will interpret this signal as a stop signal for the "Google-Extended" crawler, which is responsible for collecting materials to train Gemini models.
What Changes in Practice?
This commitment represents a shift from "technical chaos" to "unified regulation." Instead of a publisher having to chase every tech company with a specific protocol, they will be able to send a single signal that everyone understands, reducing the technical burden and providing more precise control over digital assets. However, there is a fundamental limitation to note: this standard does not have retroactive power, meaning it will not prevent companies from using data that has already been scraped in the past; instead, it will only act as a barrier for new content in the future.
Implementation Timeline
Companies vary in the speed of adopting this standard; Anthropic has already begun supporting it, while OpenAI has pledged to implement it within the next few months. As for Google and Microsoft, their actual support is expected to begin in early 2027, with Microsoft setting February 1, 2027, as the date to start support.