Big AI Labs Back One robots.txt Switch to Opt Out of Training
Website owners may soon get one switch to tell several AI companies “don’t train on my pages.” OpenAI, Google, Microsoft and Anthropic say they will honor a shared signal in robots.txt, the small file sites already use to talk to crawlers.
What’s confirmed. Today, a site has to write separate rules for each AI crawler. With the new signal, one line would reach every company that honors it. The Verge reports that all four say they will keep their AI crawlers off sites that use it. The proposal was written by a standards lead at Anthropic as part of the Internet Engineering Task Force (IETF), the group that sets many web standards.
The timing. Only Anthropic has adopted it so far, per The Verge. Google and Microsoft plan support next year. OpenAI only commits to the coming months. Meta voiced support but did not say it will honor the signal.
Search stays separate. Microsoft says it will offer finer controls, letting sites opt out of AI training or AI grounding while still showing up in Bing search. Publishers also have other tools, like the Really Simple Licensing standard for charging crawlers and Cloudflare’s default blocking of AI bots.
Our take. Less busywork for publishers is a real win. But this is voluntary, it only covers companies that sign on, and it does nothing about content already scraped. A robots.txt line is a polite request, not a lock. Publishers who want enforcement still need blocking or licensing tools.
What’s real here: a simpler opt-out is coming, but it runs on the honor system and most labs have not turned it on yet.
Sources
- OpenAI: Making it easier for websites to express AI training preferences
- The Verge: The biggest names in AI are backing a new way to opt out of scraping
By Willa Ravenwire · AI Bandwagon