Four AI companies have agreed to honour a single AI training opt-out written into robots.txt. OpenAI, Google, Microsoft and Anthropic each said on 30 October 2026 that they will read a shared policy token named __usage_ai-training. Only Anthropic is reported to support it now; the rest arrive in 2027.
What the Shared Opt-Out Actually Is
It is a new way to write one AI training preference that several companies agree to read, instead of maintaining a separate rule for each crawler. Until now a site owner wanting to refuse AI training had to name each company’s bot individually, and keep that list current as new ones appeared.
The mechanism comes from an Internet-Draft published at the Internet Engineering Task Force on 30 October 2026, titled Machine-Readable Training Preference Signal, written by Kevin Kelley of Anthropic and filed with the IETF’s AI Preferences working group. It is a Standards Track draft that expires on 3 May 2027, and the document states plainly that it is a work in progress that may change at any time.
The draft adds no new syntax to robots.txt. It reuses the existing file format defined in RFC 9309 and simply reserves a token that stands for a usage preference rather than for a crawler.
The Exact Robots.txt Lines
The draft’s own example is three lines. A site owner writes an ordinary robots.txt group whose User-Agent line names the policy token instead of a bot.
| Robots.txt line | What it expresses |
|---|---|
| User-Agent: __usage_ai-training | Opens the group that states the AI training preference |
| Disallow: / | A preference against AI training across the whole site |
| Allow: /press-kit/ | A preference in favour of AI training for paths under /press-kit/ |
Path matching follows the normal robots.txt rules, including longest-match precedence, so a narrower Allow line overrides a site-wide Disallow. A single group may name several policy tokens, and tokens can sit alongside ordinary crawler names.
Two details catch people out. A wildcard group, User-Agent: *, does not count as matching the policy token. And if a robots.txt file contains no __usage_ai-training group at all, the draft treats that as expressing no preference either way, rather than as consent.
A second and separate draft at the same working group, draft-ietf-aipref-attach-05, written by Google’s Gary Illyes and Mozilla’s Martin Thomson and dated 19 August 2026, defines a different robots.txt directive for the same purpose, written as Content-Usage: train-ai=n. The token adopted this week is not that one. Both remain drafts, and the IETF has ratified neither.
Who Honours It, and From When
Three of the four commitments are dated 2027. Nothing about the announcement changes how any site is treated today.
| Company | When it starts honouring the token | What the company said |
|---|---|---|
| Anthropic | Reported as already implementing | Authored the draft; its support is reported by trade outlets rather than set out in a dated company commitment |
| OpenAI | “The coming months”, no date | Says the signal will sit alongside, not replace, its existing robots.txt controls, and that it will publish implementation details later |
| Microsoft (Bing) | 1 February 2027 | Begins honouring the new preferences as it crawls pages |
| 2027 | John Mueller wrote that Google aims “to support this new control along with other AI providers in 2027”; its crawler note says early 2027 |
Google added one clarification that matters for anyone already using its company-specific control. If a site uses the shared token to opt out of training and does not separately name Google-Extended, Google says it plans to interpret that as if the publisher had disallowed Google-Extended. Company-specific controls continue to work alongside the shared signal.
What Microsoft Is Doing Differently
Microsoft is splitting the single question into three, so that refusing training does not automatically remove a site from AI answers. Krishna Madhavan, Principal Product Manager at Microsoft Bing, described the change as “newer, simpler, and clearer preferences that allow you to make choices about training, grounding, or both”.
- AI Training: Allows or blocks use of the site’s content to train AI models.
- AI Grounding: Allows or blocks use of the content to help generate AI answers.
- Limited Grounding: Lets the content support Microsoft’s own AI products but not its third-party grounding services.
Microsoft’s own timetable runs longer than the single February date suggests, and it carries a deadline that can cost an unwary publisher the protection it already has.
- 15 December 2026: Temporary grounding controls appear in Bing Webmaster Tools for eligible sites already using the older NoArchive or NoCache directives.
- 1 February 2027: Microsoft begins honouring the new preferences as it crawls pages.
- 1 March 2027: Publishers relying on NoArchive or NoCache must configure the new preferences by this date, after which those tags stop counting as a training or grounding preference.
- 30 September 2027: The temporary grounding preferences expire.
Microsoft says these choices do not affect a site’s eligibility for ordinary Bing search results, though blocking grounding can reduce how often a site appears in AI answers.
What the Signal Does Not Do
It is a preference, not a lock. The draft is explicit that it defines only how a preference is expressed and read, and leaves compliance to the companies consuming it. No enforcement mechanism is described.
- It is not a crawling directive: The token states what may be done with content, not whether a bot may fetch it. Crawl rules remain separate.
- It does not bind anyone outside the four: Any operator that has not agreed to read the token is unaffected by it.
- It is not retroactive: Nothing in the signal reaches content already collected, or models already trained on it.
- Private agreements can override it: The draft notes that specific arrangements generally take precedence over general expressions.
As of 31 October 2026 the signal is an unratified IETF draft, and the only reported production support is Anthropic’s. This is a voluntary commitment by four companies to read a file they were already reading, on a schedule they set themselves. It follows a year in which publishers found that opting out was often not offered at all, and that company-specific clauses could be dropped while the crawlers kept running.
What Site Owners Should Do Now
- Decide the preference first, separately for training and, if Bing matters to you, for grounding. They are now different questions.
- Add the __usage_ai-training group to robots.txt if you want to refuse training. It is inert until each company switches on support, and harmless in the meantime, because parsers ignore groups naming an unrecognised user agent.
- Keep any existing company-specific rules in place. The shared token does not replace Google-Extended, OpenAI’s controls or any other named bot, and all four companies say the older controls still work.
- If you rely on NoArchive or NoCache for Bing, diarise 1 March 2027. After that date those tags no longer count as a training or grounding preference.
- Do not treat the file as a block. If the goal is to stop fetching rather than to state a preference, that remains a job for crawler rules, authentication or network-level controls.
Frequently Asked Questions
What Is the AI Training Opt-Out in Robots.txt?
It is a shared policy token, __usage_ai-training, that a site writes into robots.txt to state whether its content may be used to train AI models. OpenAI, Google, Microsoft and Anthropic said on 30 October 2026 that they will honour it.
What Exactly Do I Add to Robots.txt?
A group beginning with the line User-Agent: __usage_ai-training, followed by Disallow: / to refuse training across the site. Allow lines can carve out paths where training is permitted, using normal longest-match precedence.
Does the AI Training Opt-Out Stop AI Crawlers Visiting My Site?
No. The IETF draft states that the token is not a crawling directive. It expresses what may be done with content after it is fetched, and crawler access rules remain a separate matter.
Does It Remove My Content From Models Already Trained on It?
No. The signal applies going forward only. It has no effect on content already collected or on models that have already been trained.
Will Opting Out of AI Training Remove My Site From Search?
Microsoft says the preferences do not affect eligibility for ordinary Bing search results, and Google keeps its training controls separate from Search indexing. Blocking grounding, however, can reduce how often a site is cited in AI answers.
