🔍 Read the full analysis: The AI Training Opt-Out Explained: Coverage, Gaps, And Setup on ThorstenMeyerAI.com
Get business pricing on office and shipping supplies
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
OpenAI, Google, Microsoft and Anthropic have announced plans or support for a shared robots.txt signal asking them not to use a website’s content to train generative AI models. The signal is voluntary, does not block crawling or AI answers, and will take effect on different schedules; Meta has expressed support without committing to honor it.
OpenAI, Google, Microsoft and Anthropic have announced plans or support for a shared website signal asking them not to use a publisher’s content to train generative AI models. The rule, published as an IETF Internet-Draft on October 30, offers website operators one line to express that preference, but it is voluntary and does not block crawling or the use of live web content in AI answers.
The proposed robots.txt group uses User-Agent: __usage_ai-training followed by Disallow: /. It is described in the draft “Machine-Readable Training Preference Signal,” written by Anthropic standards lead Kevin Kelley. The draft is an individual submission, not an adopted internet standard, and the companies’ announcements do not mean they will all implement it at the same time.
Anthropic says it has started implementation, while noting it will take time to add support across its services. OpenAI says it aims to adopt the signal “in the coming months.” Google’s documentation gives an “early 2027” target. Microsoft says it will honor the training preference starting February 1, 2027, as it crawls websites. Meta’s communications chief expressed support for publishers’ ability to state preferences but did not commit the company to honoring this signal.
Existing crawler-specific controls, including rules for GPTBot, Google-Extended and ClaudeBot, remain in place, according to the companies’ statements. Microsoft is also introducing a separate page-level control for “AI Grounding,” its term for content used to inform AI answers in Bing, Copilot and Microsoft 365. Microsoft says its new controls replace NoArchive and NoCache as training or grounding preferences after March 1, 2027.
Infographic 01 · Who backs it, and when
One line in robots.txt, four companies, four timelines.
A shared signal against AI training. Only Anthropic says it has started; the others name dates or months. The standard itself is still a draft.
- 30 Oct 2026Draft published, four companies announce supportdoneIndividual Internet-Draft by Kevin Kelley (Anthropic)
- 30 Oct 2026Anthropic: “has already started implementing support”startedIts standards lead on the IETF list: “It’ll take us some time to add support across our services.”
- Coming monthsOpenAI aims to implement itplanExisting GPTBot and OAI-SearchBot rules keep applying
- 16 + 19 Nov 2026Working group meets at IETF 127, San FranciscoopenOn 21 Oct a chair said no consensus has been declared yet
- 1 Feb 2027Microsoft honors the new preferences as it crawlsplanTraining in robots.txt or per page; grounding per page
- 1 Mar 2027Bing stops reading NoArchive / NoCache as AI preferencesdeadlinePublish the new settings before this date
- 2027Google plans support; its docs say “early 2027”planA “no” maps to Google-Extended unless you set that token yourself
- 3 May 2027Draft version 00 expires unless updatedInternet-Drafts are valid for at most six months
Binding? No, voluntary
- OpenAI calls it a “voluntary effort”.
- The vocabulary draft does not “ensure that preferences are followed”.
- The draft “has no formal standing in the IETF standards process”.
- Private agreements may override it, the draft says.
Who else?
- Meta supports a common signal, but did not say whether it will honor it.
- Apple, Amazon, xAI, Perplexity: no statement in the sources reviewed for this article.
- The draft’s acknowledgments list contributors “for identification only”, not supporters.
Sources: IETF draft, 30 Oct 2026 · Anthropic on the IETF list · AIPREF minutes, 21 Oct 2026 · Google, 30 Oct 2026 · Google crawler docs · OpenAI, 30 Oct 2026 · Microsoft Bing, Oct 2026 · Meta (Andy Stone on X). Company plans are company statements.
Powered by Thorsten Meyer AI
The proposal could reduce the work involved in communicating a training preference. Until now, publishers have had to account for different company-specific crawler names and controls. A shared rule could give participating services a common, machine-readable request, while leaving existing provider-specific instructions useful during the rollout.
But a robots.txt instruction is not an access-control system. The signal communicates a publisher’s preference; it does not technically prevent a crawler from retrieving a page, and the announcements do not establish a universal enforcement mechanism. Publishers should not treat the new line as a guarantee that content will be excluded from training by every AI company.
The boundary between training and other uses also matters. This signal addresses model training, not whether a service can use current web pages to answer a user’s question. Microsoft’s separate AI Grounding control illustrates that those are distinct choices. Microsoft also says its preferences do not change eligibility for Bing search results, while Google says Google-Extended does not affect Google Search. The announcement creates no payment entitlement: Microsoft said neither choice guarantees inclusion, traffic or payment.
Popular right now
Ninja DualZone Smart XL Air Fryer, 10-Qt, 6-in-1, DZ950
Bose QuietComfort True Wireless Noise-Cancelling Earbuds, Black
Vitamix Propel Series 510 Professional-Grade Blender, 3 Programs, Black
As an affiliate, we earn on qualifying purchases.
From Separate Crawlers to One Draft
Website operators have long been able to use robots.txt to give crawlers instructions, but AI companies have used separate identifiers and rules. Publishers wanting to limit model-training use therefore needed to check each provider’s guidance and configure its controls separately. The proposed shared token is intended to express the training preference across participating companies with one rule.
The document builds on work by the IETF’s AI Preferences working group, which considers ways to express machine-readable preferences. However, publication as an Internet-Draft does not make the text an official standard. At a working-group meeting on October 21, one of the group’s chairs said, “We haven’t declared consensus on anything yet.” The draft’s author is affiliated with Anthropic, a company that has said it is implementing the signal; that connection is relevant context, not evidence that the draft has been adopted by the IETF.
The source report also discloses that its research and drafting used AI assistance, including Anthropic’s Claude, and that its publisher belongs to a network of websites affected by AI companies’ use of web content. Those disclosures describe the report’s potential interests and process; the company-by-company commitments above are attributed to their public statements and documentation.
““has already started implementing support for this draft””
— Anthropic
Coverage and Enforcement Still Vary
Company participation is incomplete. Meta has voiced support for the principle but has not said whether it will honor this particular signal. The source report found no commitment from Apple, Amazon, xAI or Perplexity in the material it reviewed; that does not establish what those companies may do later. It is also unclear how broadly participating companies will apply the rule across products and services, especially while implementations are being added.
The draft’s status could change through the IETF process, but it currently has no formal standing as an IETF standard. Its voluntary nature leaves open what recourse a publisher would have if a service ignored the preference. The rule also does not undo use of content in models that have already been trained, and it does not itself govern live retrieval for AI-generated answers. Those limits mean “opt-out” should be read as a stated training preference, not a complete prohibition on every AI use of a website.
Check Rules as Rollout Dates Approach
Publishers considering the new control can add the shared rule to robots.txt while keeping their existing provider-specific instructions until each service’s support is in place. They should review each company’s documentation as its implementation date approaches, rather than assuming that a single change immediately covers every crawler or product.
Microsoft’s dates call for particular attention: its shared training and separate AI Grounding controls are due to take effect on February 1, 2027, while NoArchive and NoCache stop counting as training or grounding preferences after March 1, 2027. Google’s stated target is early 2027, and OpenAI’s is the coming months; Anthropic says implementation has begun. Further public announcements, updates to crawler documentation and progress in the IETF working group will show whether participation expands and whether the draft changes.
Key Questions
What does the shared AI-training signal say?
It is a robots.txt rule group using User-Agent: __usage_ai-training and Disallow: /. It communicates a preference that the site’s content not be used to train generative AI models by services that honor it.
Does the rule stop AI companies from crawling a website?
No. The signal is a voluntary preference, not a technical barrier that blocks access. It also does not remove a site’s content from models that have already been trained.
Does opting out of training block AI answers using a page?
Not under this signal. It addresses model training, not the use of live web content to inform answers. Microsoft has announced a separate AI Grounding control for that distinct purpose.
Which companies have said they will honor it?
Anthropic, OpenAI, Google and Microsoft have announced implementation plans or support, with different timelines. Meta expressed support for publishers’ ability to state preferences but did not commit to honoring this specific signal.
Is the signal an official internet standard?
No. It is an Internet-Draft associated with the IETF AI Preferences working group. The source report says it has no formal standing in the IETF standards process, and the group had not declared consensus.
Source: ThorstenMeyerAI.com
