OpenAI announced it will honor a new shared signal in robots.txt — the plain-text file websites use to set rules for automated crawlers — that lets site operators state their AI-training preferences once instead of maintaining separate opt-outs for every individual crawler. The move, detailed in OpenAI’s post “Making it easier for websites to express AI training preferences”, draws on a draft standard from the IETF’s AI Preferences working group. Google and Microsoft Bing published matching commitments the same day, and Anthropic’s Kevin Kelley is the author of the underlying technical draft, making this a rare moment of coordinated action across four major AI labs at once.
Why this matters
Until now, a publisher that wanted to block AI training had to track a growing list of crawler names — GPTBot, Google-Extended, ClaudeBot, Applebot-Extended, CCBot and more — and keep separate rules current for each one as new bots appeared. A single shared token collapses that into one line. For small publishers and site operators without dedicated legal or technical teams, that’s a meaningful reduction in busywork, even if it doesn’t resolve the deeper disputes over compensation and consent that have driven lawsuits against OpenAI and others.
It’s also notable as a rare example of direct, voluntary coordination between competitors on a shared technical standard, rather than each lab building its own walled-garden crawler rules. That kind of interoperability is genuinely good for the open web, and it’s the sort of unglamorous infrastructure work that rarely gets headlines but makes everyone’s lives easier if it’s actually adopted widely.
The caveats are worth being clear-eyed about. This is a voluntary signal with no enforcement mechanism — nothing stops a non-participating crawler, or a bad actor, from ignoring it entirely. OpenAI has given no firm implementation date beyond “coming months,” and the underlying IETF draft is still just that: a draft, not a ratified standard. It also doesn’t retroactively address content already used to train existing models, and it sits alongside — rather than replacing — each company’s existing crawler-specific controls, so publishers will still need to understand both systems for a while. Useful plumbing, not a resolution to the training-data fight.
Leave a Reply