On October 30, 2026, four of the biggest names in artificial intelligence said they will back one shared robots.txt instruction for publishers who want to opt out of AI training. OpenAI, Google, Microsoft and Anthropic agreed on the same signal, and the news was covered within a day by SiliconANGLE, PPC Land and The Indian Express. For publishers, the promise is simple: one line, understood by all four companies, instead of a separate rule for every crawler.
What the signal is
The shared token is __usage_ai-training, paired with a Disallow rule. Instead of naming a specific bot, it names a purpose: training. That means the instruction survives crawler renames, because it does not depend on a bot keeping the same name. Until now, publishers had to list GPTBot, Google-Extended, ClaudeBot and others one by one, and update that list whenever a company renamed or added a crawler. The idea is written up as IETF Internet-Draft draft-kelley-aipref-training-preference-signal-00, authored by Kevin Kelley of Anthropic as an individual submission, expiring May 3, 2027. It is a proposal the four companies agreed to honor, not yet an IETF standard.
Who honors it and when
The four companies are not moving at exactly the same speed. Anthropic is already implementing the signal. OpenAI says support will arrive in the coming months. Microsoft is reported by PPC Land to follow on February 1, 2027, a date reported, not confirmed by Microsoft in the material used here. Google is expected in 2027, with no specific date given. Meta has expressed support, but has made no implementation commitment, so publishers should not count on Meta yet.
What it does not cover
The signal is narrow, and publishers should not over-read it. It does not cover content already collected or training already done, it does not block crawling itself, and it does not change search rankings or non-generative models. There is also a split on live answers: Google treats the token as blocking both training and Gemini grounding, while Microsoft limits it to training and handles grounding separately via an ai-grounding meta tag. A wildcard User-Agent: * group does not trigger the preference, so the token must be named explicitly. Smaller labs and open-weight builders are not bound by it, and robots.txt has always been advisory, not a technical block.
Why it matters
One line replaces a maintenance chore, and it gives publishers a documented refusal that strengthens licensing arguments. The short take is clear: this protects future crawls only, not the past. Publishers must weigh control against visibility in AI answers, because opting out may mean less presence where readers now find information. Still, a single shared signal is a step forward from four separate lists, and it is worth adopting while the details are worked out.
Comments
Post a Comment