For years, publishers have had to play whack-a-mole with AI crawlers — a different block for every bot, every company, every model. On Friday, October 30, 2026, four of the biggest AI companies said they will finally fix that part: OpenAI, Google, Microsoft and Anthropic pledged a single, standardized way for websites to say "do not scrape this to train AI." One instruction, recognized by all of them. It is the closest the AI industry has come to a shared opt-out — and it comes after years of lawsuits, protests and publishers watching their work feed models for free.
What the pledge actually promises
Today, a publisher that wants out of AI training has to implement multiple blocking commands, one per crawler. The new plan replaces that mess with one standardized instruction that any adopting AI company honors. Enter it once, and OpenAI, Google, Microsoft and Anthropic should all respect it for future training. Meta has also voiced support for the effort through spokesperson Andy Stone, who called a common signal "an important step." The details of the standard itself are still being worked out across the industry, so this is a pledge and a direction, not a finished switch you can flip today.
When it actually arrives — and the timeline gap
This is where the excitement needs a reality check. The switch will not fully take effect for months. Microsoft says it will implement it beginning February 1, 2027, and Google sometime in 2027. OpenAI says it will roll it out "in the coming months," while Anthropic says it has "already started" rolling it out. In other words, the companies have agreed on the destination, but they are arriving at very different speeds — and until each one ships its version, publishers are still stuck with today's patchwork of blocks.
The catches publishers should know
First, scraping stays on by default. Nothing changes unless a publisher actively turns the new setting on. Second, the setting covers future training only — it does not undo training on content already collected, and it does not stop bots from crawling a site for other purposes. Third, opting out may have a cost: Microsoft and Google both warned that opting out of AI training could reduce the chances that a publisher's content surfaces in AI apps and experiences. Publishers are being asked to trade visibility for control, with no clear way yet to measure either side of that deal.
Why this fight got here
Publishers have been raising the alarm for years as bots surged across the web. The New York Times sued OpenAI and Microsoft in December 2023, alleging millions of its articles were used without authorization to train ChatGPT; the companies have invoked fair use. Separately, OpenAI
paused training of its most powerful models in September after a security incident. Cloudflare has blocked AI bots by default on new customer sites since July 2025. And this is not the first time a big platform has tried to quietly rewrite the rules on training data —
Meta's own terms clause drew a backlash earlier this month before it was removed. This joint pledge is, in part, the industry trying to show it can
self-regulate before regulators do it for them.
AiPost's take
A single opt-out is genuinely good news — the current system is so fragmented that only the biggest publishers can afford to police it properly. But a pledge is not a product, and "in the coming months" is doing a lot of heavy lifting here. Watch three things: whether smaller AI companies adopt the same standard, whether the switch is simple enough for a small blog or local news site to use, and whether opting out really does hurt visibility in AI answers. If all three go the right way, this could be the moment the web gets a real "do not train" button. If not, it will be remembered as a very well-worded press release.
Comments
Post a Comment