OpenAI said on October 30, 2026 that it will honor one shared signal in robots.txt that lets a website say no to generative AI training once, for every participating provider, instead of writing rules for each company’s crawler. Google and Microsoft said the same day that they will support it too. The signal is a policy token, __usage_ai-training, defined in an Internet-Draft written by Kevin Kelley of Anthropic and posted to the IETF the same day.
None of it is live yet. OpenAI says it aims “to implement the signal in the coming months”. Google’s crawler documentation says it intends to support it in early 2027, and Microsoft is firmest: “Starting February 1, 2027, Microsoft will honor the new preferences as it crawls.” Anthropic, whose employee wrote the draft, had posted no announcement on its newsroom when we checked at 18:33 UTC.
We read OpenAI’s announcement, the draft itself, Microsoft Bing’s blog post and Google’s common crawlers page.
On this page · 6 sectionsOpen
OpenAI said on October 30, 2026 it will honor one shared robots.txt signal for AI training preferences; Google and Microsoft said the same.
The signal is a robots.txt group whose User-Agent line is the policy token __usage_ai-training, defined in a draft by an Anthropic engineer.
A wildcard group does not count, and a file without the token expresses no training preference at all.
It controls training, not crawling: to stop a crawler fetching pages, a site still names that crawler.
Nothing is live yet: OpenAI says the coming months, Google early 2027, Microsoft February 1, 2027.
§ 01Who has said what
| Provider | What it said | When |
|---|---|---|
| OpenAI | Will honor the shared signal; its existing crawler controls stay | In the coming months |
| Intends to support it with user-agent __usage_ai-training | Early 2027 | |
| Microsoft | Joins it, adds page-level training and a separate grounding preference | February 1, 2027 |
| Anthropic | Its employee wrote the draft; no company announcement found | Not stated |
OpenAI calls it a voluntary effort and ends its post with a nudge to the rest of the industry: “We encourage other AI providers to honor the signal.”
§ 02How the signal works
The draft adds no new directive and no new grammar. A site writes an ordinary robots.txt group whose User-Agent line names the policy token instead of a crawler. This is the draft’s own example:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
That says no to AI training on every path except /press-kit/, where it says yes. Three rules in the draft decide how a provider reads it:
- Wildcard groups do not count. A
User-Agent: *group “MUST NOT be interpreted as matching the policy token”, so a site that blocks everything with a wildcard has not said anything about training. - No group, no preference. If the file has no
__usage_ai-trainingline, then “no preference has been expressed.” - It does not control crawling. “This policy token is not a directive for crawling behavior.” A provider may still fetch a page it should not train on. To stop the fetch itself, a site still names the crawler, as it does today.
§ 03What to write for each goal
| You want | What to write |
|---|---|
| No AI training anywhere, still in search | A __usage_ai-training group with Disallow: / |
| Training allowed except some folders | A __usage_ai-training group that disallows those paths |
| A crawler never fetches your pages | A group naming that crawler, such as GPTBot or ClaudeBot |
| Microsoft never grounds answers on a page | Microsoft’s page-level grounding preference |
OpenAI gave no date beyond the coming months, so it is not on the timeline.
§ 04Microsoft goes further
Microsoft’s post adds two things the draft does not have. Its training preference also works per page, and a page-level setting overrides robots.txt. And it adds a separate grounding preference for whether a page can inform AI answers in Bing, Copilot and Microsoft 365: Disallow stops grounding, and a new DisallowExtended stops it only in services Microsoft sells to others, such as grounding through its API. Both default to allowed. The new settings replace the NoArchive and NoCache tags, which Microsoft stops reading as training or grounding preferences after March 1, 2027. It is plain about the trade: “Neither choice guarantees inclusion, traffic, or payment.”
§ 05What is not settled
- It is an individual draft. The IETF datatracker says the document is “not endorsed by the IETF” and has no formal standing yet. The working group’s vocabulary draft is still in progress.
- Company tokens may win. The draft says “Specific expressions SHOULD take precedence over general ones”, and lets a provider treat its own token, such as a GPTBot group, as more specific than the shared one.
- Deals override it. “Private agreements MAY override preferences expressed via this mechanism.”
- Anthropic itself. Its employee wrote the draft and its help page still describes opting out through a ClaudeBot rule. We found no Anthropic statement on honoring the new token.
- Everyone else. Meta, Apple and Perplexity were not part of the October 30 announcements we could read.
§ 06Where CellCog stands
Our conflict, declared: CellCog publishes a blog and product pages that we want AI engines to read and cite. Our robots.txt has allowed every crawler, training crawlers included, since September 2, 2026, and it has no __usage_ai-training group, so under the draft it expresses no preference at all. For a site that does want a say, the change is real: one group instead of a list of company names, read by three of the largest providers within a few months.
Q1What do I add to robots.txt to opt out of AI training?
A group with User-Agent: __usage_ai-training and Disallow: / for the whole site, or Disallow lines for the paths you want kept out of training.
Q2Does User-Agent: * cover it?
No. The draft says a wildcard group must not be read as matching the policy token, so it has to be named.
Q3Does Anthropic honor it?
An Anthropic engineer wrote the draft, but we found no Anthropic statement on honoring the token as of October 30, 2026.
Q4What does Microsoft add?
A page-level training preference that overrides robots.txt, and a separate grounding preference with Disallow and a new DisallowExtended.
Q5Will opting out hurt search rankings?
Microsoft says its new preferences do not change eligibility for Bing search results, and Google keeps training controls separate from Search.