Major AI Labs Adopt Unified Standard for Web Publishers to Block Model Training
OpenAI, Google, Microsoft, and Anthropic have committed to a unified web standard allowing publishers to opt out of AI scraping through a single policy token.
Four of the largest artificial intelligence developers—OpenAI, Google, Microsoft, and Anthropic—have agreed to adopt a unified standard that lets web publishers block automated AI scraping through a single control, according to The Verge. The new framework allows site owners to broadcast their data training preferences across participating AI companies simultaneously, replacing the tedious requirement of configuring unique exclusion rules for every individual corporate crawler.
The shared policy token was proposed within the Internet Engineering Task Force by Kevin Kelley, a technical lead for standards at Anthropic, as reported by Search Engine Roundtable and confirmed by The Verge. The signal will reside inside a website's traditional robots.txt file, which has historically regulated how automated web bots index content for search engines, archiving services, and machine learning models.
Under current web administration practices, digital publishers must maintain individual directives for every AI company operating a crawler on their servers. The unified protocol arrives as online media outlets struggle to manage an influx of automated scraping bots feeding content into large language models. Some industry groups have tried alternative solutions to regain leverage, such as the Really Simple Licensing standard for charging visiting bots, while network providers like Cloudflare have moved to block AI scrapers by default.
Under the new arrangement, OpenAI, Google, Microsoft, and Anthropic have pledged to restrict their respective AI crawlers from accessing web pages whenever their systems detect the shared policy token. Meta communications head Andy Stone also voiced support for the joint signal, describing it as a vital step forward, though he did not explicitly confirm whether Meta will honor the token across its own infrastructure.
Participating search vendors are taking care to separate model training permissions from search indexing. Microsoft stated that it will offer granular options permitting site owners to opt out of either AI training or AI grounding—the retrieval technique that directs models to verified sources to limit hallucinations—without losing visibility in Bing search results. Google previously introduced similar settings that separate AI training and AI search capabilities from traditional search ranking.
Implementation schedules vary among the frontier labs. According to The Verge, Anthropic has already implemented the policy token signal across its operations. OpenAI expects to launch its integration in the coming months, while Google and Microsoft plan to fully support the new standard next year.