OpenAI, Google, Microsoft and Anthropic say they will honor a shared robots.txt "policy token" for opting out of AI training scraping

OpenAI, Google, Microsoft and Anthropic have said their AI crawlers will stay off sites that carry a shared "policy token" in robots.txt, signalling an opt-out from AI training. The token was proposed through the IETF by Anthropic's Kevin Kelley. Only Anthropic honours it today. Google and Microsoft say they will support it next year, and OpenAI says "coming months". Meta voiced support but did not commit. It is still a proposal, not a finalised IETF standard, and the coverage links to neither the draft nor the token syntax.

Publishers currently have to signal AI training preferences to each company separately. One token would cut that to a single line in a file they already maintain, if the commitments are kept. Publishers cannot rely on it yet, because three of the four firms have not shipped support and Meta has not committed. Microsoft's planned split between AI training and AI grounding could matter more in practice, since it would let publishers refuse training while still allowing their content to be used in answers.

Describing this as the biggest names "backing" an opt-out overstates it: only one of the four honours it now, and the rest have loose, unenforceable timelines. The token covers training preferences and relies on voluntary compliance, so it does nothing about crawlers that ignore robots.txt, and it does not settle licensing or payment, which approaches such as Really Simple Licensing address. The coverage also gives no evidence on how well companies have honoured existing robots.txt rules.

1. 01Publication of the IETF draft with the exact token syntax, and whether it moves from individual proposal to a working group item
2. 02Google, Microsoft and OpenAI actually shipping crawler support, checked against their published crawler documentation and the 'next year' and 'coming months' dates
3. 03Server log or Cloudflare-style crawler data showing whether the four firms' bots stop fetching pages that carry the token, and whether Meta commits