🔍 Read the full analysis: AI Training Opt-Out: What’s Included, What’s Not, And How To Set It on ThorstenMeyerAI.com
Get the little things that make your day delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
OpenAI, Google, Microsoft and Anthropic have announced support for a shared robots.txt signal asking that a site’s content not be used to train generative AI models. The signal is voluntary, does not block crawling or AI answers based on live web content, and will roll out on different schedules; Meta has expressed support but made no implementation commitment.
OpenAI, Google, Microsoft and Anthropic said on October 30 that they will honor a shared robots.txt signal asking that a website’s content not be used to train generative AI models. The move gives publishers one proposed instruction to use across participating services, but it is a voluntary request, not a technical block, and each company has a different implementation timeline.
The signal appears in an Internet-Draft called “Machine-Readable Training Preference Signal”, published by Anthropic standards lead Kevin Kelley. The proposed robots.txt group uses User-Agent: __usage_ai-training followed by Disallow: /. A site owner who adds it is expressing a preference against training use; the rule does not prevent a crawler from accessing the site.
Anthropic says it has started implementing support, though it says adding the signal across its services will take time. OpenAI says it aims to honor the signal “in the coming months.” Google says it aims to support it in 2027; its crawler documentation specifies “early 2027.” Microsoft says its shared training preference takes effect as it crawls sites from February 1, 2027.
Microsoft is also introducing a separate page-level preference for AI Grounding—use of web content to inform answers in Bing, Copilot and Microsoft 365. Microsoft says both new preferences take effect February 1, 2027. Its older NoArchive and NoCache tags will stop counting as training or grounding preferences after March 1, 2027. Google says that if a site uses the shared signal without a separate Google-Extended rule, it plans to treat that as a disallow for Google-Extended.
Infographic 01 · Who backs it, and when
One line in robots.txt, four companies, four timelines.
A shared signal against AI training. Only Anthropic says it has started; the others name dates or months. The standard itself is still a draft.
- 30 Oct 2026Draft published, four companies announce supportdoneIndividual Internet-Draft by Kevin Kelley (Anthropic)
- 30 Oct 2026Anthropic: “has already started implementing support”startedIts standards lead on the IETF list: “It’ll take us some time to add support across our services.”
- Coming monthsOpenAI aims to implement itplanExisting GPTBot and OAI-SearchBot rules keep applying
- 16 + 19 Nov 2026Working group meets at IETF 127, San FranciscoopenOn 21 Oct a chair said no consensus has been declared yet
- 1 Feb 2027Microsoft honors the new preferences as it crawlsplanTraining in robots.txt or per page; grounding per page
- 1 Mar 2027Bing stops reading NoArchive / NoCache as AI preferencesdeadlinePublish the new settings before this date
- 2027Google plans support; its docs say “early 2027”planA “no” maps to Google-Extended unless you set that token yourself
- 3 May 2027Draft version 00 expires unless updatedInternet-Drafts are valid for at most six months
Binding? No, voluntary
- OpenAI calls it a “voluntary effort”.
- The vocabulary draft does not “ensure that preferences are followed”.
- The draft “has no formal standing in the IETF standards process”.
- Private agreements may override it, the draft says.
Who else?
- Meta supports a common signal, but did not say whether it will honor it.
- Apple, Amazon, xAI, Perplexity: no statement in the sources reviewed for this article.
- The draft’s acknowledgments list contributors “for identification only”, not supporters.
Sources: IETF draft, 30 Oct 2026 · Anthropic on the IETF list · AIPREF minutes, 21 Oct 2026 · Google, 30 Oct 2026 · Google crawler docs · OpenAI, 30 Oct 2026 · Microsoft Bing, Oct 2026 · Meta (Andy Stone on X). Company plans are company statements.
Powered by Thorsten Meyer AI
A common instruction could reduce the work publishers face when expressing training preferences to multiple AI companies. Until now, site operators have generally needed company-specific crawler rules, such as GPTBot, Google-Extended and ClaudeBot, and Microsoft-specific page tags. Under the announced approach, one shared line could communicate a training preference to several participating providers.
But the practical effect depends on companies implementing and honoring the request. The signal does not force compliance, and it does not itself provide a way to remove material from models that have already been trained. It also does not guarantee payment, search visibility or reduced crawling. Microsoft says its preferences do not change eligibility for Bing search results; Google says Google-Extended does not affect Google Search.
The distinction between training and other uses matters to publishers deciding what to allow. A site may want to restrict model training while still permitting search indexing, or may have separate views about whether its pages can inform AI-generated answers. Microsoft’s separate grounding control makes that distinction explicit for its services; the shared signal described in the draft addresses training, not every use of website content.
Products Worth Considering
Ring Alarm 8-Piece Kit (2nd Gen) Home Security System
eufy Security Essential 7-Piece Alarm System
Ring Alarm 5-Piece Wireless Security System
As an affiliate, we earn on qualifying purchases.
How the Signal Is Being Developed
The proposal is connected to the AI Preferences working group at the Internet Engineering Task Force, an organization that develops internet standards. The companies’ announcements refer to the same individual Internet-Draft, but that does not make the signal an adopted standard. The draft itself says it has no formal standing in the IETF standards process, and public discussion is still underway.
At an October 21 working-group meeting, one of the group’s chairs said, “We haven’t declared consensus on anything yet.” That status is relevant because the announced support is company-level intent for a proposal still under discussion, rather than confirmation that an established internet standard has taken effect.
Existing company-specific controls remain relevant during the rollout. OpenAI says its current crawler controls will continue to apply. Google’s announced fallback for sites without a Google-Extended rule is specific to its interpretation of the new signal. Anthropic’s crawler help page, as of October 31, still described its existing per-crawler controls.
““has already started implementing support for this draft””
— Anthropic standards lead Kevin Kelley
Adoption and Coverage Still Vary
Meta has not confirmed it will honor the signal. Its communications chief, Andy Stone, said the company supports publishers’ ability to express preferences and called a shared signal an important step, but did not make an implementation commitment. The source material reports no statements from Apple, Amazon, xAI or Perplexity about adopting it.
It also remains unclear how the proposal’s wording and implementation may change as the IETF working group discusses it. Company statements set different target dates and describe support at different stages; those announcements do not establish that every service operated by each company will apply the preference at the same time. The rule’s status as a request means the extent of compliance cannot be guaranteed by the file itself.
For website owners, the boundaries matter: the shared instruction concerns training, not content retrieved from the live web to compose an answer. Nor does it undo prior training. The material provided does not establish a universal mechanism for removing already-used content or resolving disagreements about whether a particular use counts as training.
Rollout Dates and Site Controls
Publishers considering the rule can add the proposed User-Agent: __usage_ai-training group with Disallow: / to robots.txt, while retaining existing company-specific rules during the transition in implementation. They should check each provider’s documentation and announcements rather than assume that adding the shared line immediately changes every crawler’s behavior.
Microsoft’s published schedule calls for its new training and grounding preferences to apply from February 1, 2027, as it crawls websites. Sites using NoArchive or NoCache for training or grounding preferences should account for Microsoft’s stated March 1, 2027 cutoff for those tags. Google’s stated aim is early 2027, while OpenAI has given a less specific “coming months” timeline. Anthropic says implementation has begun but has not provided a complete service-by-service schedule in the material cited.
The next developments to watch are the IETF working group’s discussion of the draft, updated crawler documentation, and company confirmations of when their services honor the signal. Until those arrive, website operators must treat the shared rule as an additional preference to monitor—not a replacement for all existing controls or a guarantee about how content will be used.
Key Questions
What should a website add to opt out of AI training?
The proposed robots.txt group is User-Agent: __usage_ai-training followed by Disallow: /. It expresses a preference against training use by companies that honor the signal; it does not technically block access to the site.
No. The proposed signal is a training preference, not a crawler-access block. Existing company-specific crawler controls may still be needed for other purposes.
Does it prevent a site’s content from appearing in AI answers?
Not by itself. The shared draft addresses model training, not AI answers based on live web content. Microsoft has announced a separate AI Grounding preference for content used in answers across its named services.
When will the companies honor the signal?
Anthropic says implementation has started; OpenAI says “in the coming months”; Google aims for 2027, with its documentation saying early 2027; and Microsoft says February 1, 2027. Meta has not committed to implementation.
Is this an official internet standard?
No. The proposal is an Internet-Draft linked to IETF working-group activity, but it has no formal standing in the IETF standards process, and the group had not declared consensus as of its October 21 meeting.
Source: ThorstenMeyerAI.com
