OpenAI, Google, Microsoft and Anthropic have backed a shared signal that could give publishers one way to communicate AI-training preferences across multiple crawlers. The proposal would place a policy token inside a site’s robots.txt file, extending a familiar web standard beyond search access rules. Instead of writing separate instructions for every AI company, a publisher could indicate whether its material may be used for model training through one common mechanism. The pledge arrives as websites face a growing volume of automated collection and continuing disputes over content rights.
The proposed signal is intended to simplify a fragmented process. Publishers currently have to identify individual crawlers and maintain separate controls for different providers, while some bots may not clearly disclose how they use retrieved material. The shared token would let participating companies recognize the same preference. Anthropic says it has already started supporting the signal. OpenAI plans to implement it in the coming months, while Google and Microsoft have said their support is planned for 2027, so the practical effect will develop gradually.
The setting would not be a universal web block. Microsoft said its controls can distinguish AI training from AI grounding, in which a model is pointed to verified sources to improve answers. The proposal also does not automatically remove a site from search results. Google and Microsoft have described similar separation between search visibility and AI-use preferences. Publishers would still need to understand which signal they are selecting, monitor compliance and decide whether reduced model training exposure is worth any possible loss of visibility inside AI products.
The announcement reflects a broader attempt to move the dispute from one-off negotiations toward shared technical rules. Cloudflare has blocked AI crawlers by default on new customer sites, while other standards seek to let publishers license or charge automated visitors. The new token will matter only if large providers honor it consistently and other AI companies join. For publishers, the immediate task is to document content policies, review robots.txt management and watch implementation details before treating the signal as a complete answer to scraping, attribution or compensation concerns.
