On October 30, 2026, Google announced through its Search Central blog that it plans to support a shared AI training control in robots.txt. Microsoft Bing, OpenAI and Anthropic have also backed the proposed signal.
For website owners, this turns a broad concern into a practical decision: should AI companies be allowed to use your content to train their models?
There is no single answer for every business. Blocking training can help protect valuable content, but choosing the wrong settings may also reduce your visibility in AI-generated answers. This guide explains what has changed, how the available controls differ and what to consider before updating your robots.txt file.
Key Takeaways
- Google, Microsoft Bing, OpenAI and Anthropic have announced support for a shared robots.txt token called __usage_ai-training.
- Google expects to support the token in early 2027. Until then, Google-Extended remains its active control.
- AI training, AI grounding and search rankings are separate uses with different controls.
- A catch-all User-agent: * rule does not express an AI training preference under the current draft.
- The proposal is still an IETF Internet-Draft, so its details may change before providers implement it.
What Did Google Announce?
Google said the new control is based on work from the Internet Engineering Task Force’s AIPREF working group. It is designed to let websites communicate an AI training preference to several participating providers through one robots.txt entry.
The proposed control uses a policy token in the User-agent field of a standard robots.txt group. It does not introduce a new robots.txt directive. Search Engine Roundtable also provides an overview of the announcement.
Google plans to recognize the signal in early 2027. If a website opts out through the shared token without mentioning Google-Extended separately, Google intends to treat that preference as a Google-Extended disallow. Website owners will still be able to use company-specific controls alongside the shared signal.
Microsoft’s announcement goes further by separating AI training from AI grounding and introducing page-level preferences. Bing plans to honor the new preferences from February 1, 2027. Publishers using older NoArchive or NoCache directives will need to configure the new preferences by March 1, 2027 if they want those restrictions to continue.
Bing will offer three main choices:
- AI Training, which controls whether content may be used for model training.
- AI Grounding, which controls whether content may support AI-generated answers.
- Limited Grounding, which allows content to support Microsoft’s AI products but not third-party grounding services.
For eligible sites using older directives, temporary grounding controls are expected to become available in Bing Webmaster Tools on December 15, 2026.
Is this already an official standard?
No. The proposal remains an individual Internet-Draft called draft-kelley-aipref-training-preference-signal-00. Some details are still being discussed, so businesses should treat it as an important industry direction rather than a finalized standard.
AI Training, Grounding and Search Are Not the Same
Many website owners assume that one setting controls every AI-related use. In reality, training, grounding and search are different.
| Control | What it governs | Current status |
| __usage_ai-training | AI model training by participating providers | Google plans support in early 2027; Bing plans to honor it from February 1, 2027 |
| Google-Extended | Gemini training and grounding | Available now |
| Googlebot rules and Google’s generative AI controls | Google Search, AI Overviews and AI Mode | Available now |
| Bing AI preferences | Training, grounding and limited grounding | Planned for February 1, 2027 |
AI training uses content to adjust a model’s learned parameters. An opt-out applies to future training. It does not remove material that may already have been used.
AI grounding retrieves information from a webpage to help create a current answer, often with a citation or link. From a marketing perspective, grounding is closer to search visibility than model training.
Search indexing determines whether a page can appear in traditional search results. Google Search uses Googlebot, which is not controlled by Google-Extended. Restrictions for AI Overviews and AI Mode rely on other tools, including snippet directives and Google’s generative AI controls.
Understanding these differences is essential. A business may want to prevent model training while still allowing its pages to be cited in AI answers and found through search.
What Happens If You Do Nothing?
If your robots.txt file does not contain a __usage_ai-training group, you have not expressed a training preference through the proposed signal. A general wildcard rule does not count as an AI training opt-out under the current draft.
Leaving the file unchanged may keep your content available for covered uses. That can be appropriate for some organizations, but it should be a deliberate decision rather than an accidental default.
Should You Allow AI Training?
The right choice depends on the value of your content and how customers discover your business.
Reasons to allow training
- Your website mainly contains public marketing information intended for broad distribution.
- Your content does not include premium, proprietary or commercially sensitive material.
- You want AI models to have more information about your brand, while understanding that training does not guarantee a citation or website visit.
Reasons to block training
- You publish original research, paid editorial content, gated resources or proprietary frameworks.
- Your content gives your business a competitive advantage that you do not want incorporated into external models.
- You want a clear, documented policy governing how AI providers may use your website.
Google notes that robots.txt is recognized as a primary machine-readable method for expressing preferences under the European Union’s Code of Practice for general-purpose AI.
The middle option: block training but allow citations
For many businesses, the most practical approach may be to restrict model training while remaining available for AI grounding.
Bing says publishers can separate these choices. Google has stated that websites that explicitly allow Google-Extended may be eligible to appear in grounded Gemini responses. However, Google has not fully explained how the shared training signal will interact with every grounding scenario. Review the latest provider documentation before applying the settings in 2027.
| Option | Approach | Best suited to | Main trade-off |
| Allow training and grounding | Publish no training opt-out | Public marketing content | Content may be used for training without compensation |
| Block training but allow grounding | Add a training opt-out while keeping grounding available | Businesses with valuable content that still need AI visibility | Provider rules are still developing |
| Block training and grounding | Restrict both uses | Paywalled, licensed or highly sensitive content | Lower visibility in AI-generated experiences |
How to Opt Out Without Damaging SEO
- Open yourdomain.com/robots.txt and review the existing rules.
- Add an explicit __usage_ai-training group when the participating providers begin supporting it. Under the current draft, a wildcard group does not express this preference.
- Make a separate decision about Google-Extended. Blocking it can also affect Gemini grounding.
- Do not block Googlebot or Bingbot as a substitute for an AI training preference. Those crawlers support search visibility.
- Use Google’s generative AI controls or snippet directives if you want to limit how content appears in AI Overviews or AI Mode.
- Review the file again when providers begin enforcing the new signal in 2027.
Example robots.txt configuration
The Google-Extended lines below are commented out intentionally. Whether they should be activated depends on your training and grounding strategy.
# Shared AI training opt-out
# Google plans to support this in early 2027
User-agent: __usage_ai-training
Disallow: /
# Optional and available now
# This can restrict Gemini training and grounding
# User-agent: Google-Extended
# Disallow: /
Robots.txt errors can be difficult to detect and may affect organic traffic. A technical SEO review can confirm that search engines still have the access they need. Analytics and reporting can then show whether search or referral traffic changes after implementation.
Common Mistakes to Avoid
- Using a wildcard as the training opt-out: A general rule does not express the proposed training preference and may unintentionally restrict search crawlers.
- Assuming the change is retroactive: The opt-out applies to future training, not content that may already have been used.
- Treating AI Overviews and model training as the same use: They rely on different systems and controls.
- Applying the setting and never reviewing it: The proposal and individual provider policies may change during the 2027 rollout.
- Changing robots.txt without testing it: A formatting error can block important pages or crawlers.
What This Means for Your Marketing Strategy
Businesses should classify their content before setting a sitewide policy.
Public service pages, product information and educational articles may benefit from broad discovery and citation. Original research, licensed resources and proprietary methods may need tighter protection. A single rule for every page may not support both goals.
Search platforms are already changing what customers see before they click. We explored one example in our article about AI-written review summaries in search ads. Your AI crawler policy should support the same visibility strategy as your SEO, advertising and content programs.
How Champ360 Marketing Can Help
Champ360 Marketing uses its Precision360 Marketing process to connect technical decisions with broader marketing goals. The process begins with an audit and competitor review, followed by recommendations, implementation and performance monitoring.
Champ360 can help you:
- Identify which search bots and AI crawlers can currently access your website.
- Separate content that needs protection from content that should remain discoverable.
- Configure robots.txt and related AI controls without unnecessarily limiting search access.
- Monitor organic traffic, referrals and AI visibility after changes are made.
Talk to Champ360 Marketing about a website audit and get a clear assessment of your AI crawler settings.
The Bottom Line
The proposed shared token could make AI training preferences easier to manage, but it will not make the strategic decision for you. Separate model training from AI citations and traditional search before changing your robots.txt file.
Protect the content that carries unique commercial value while keeping useful marketing pages discoverable. Then review the configuration as Google, Bing and other providers begin supporting the signal in 2027.
FAQs
What is __usage_ai-training in robots.txt?
It is a proposed shared policy token that allows websites to tell participating AI providers whether their content may be used for model training. It appears as a User-agent entry in robots.txt and originates from an IETF Internet-Draft.
Will opting out of AI training hurt my Google rankings?
The reviewed sources do not connect AI training opt-outs with lower Google Search rankings. Google Search relies on Googlebot, which is separate from Google-Extended. Bing also says its AI preferences do not affect traditional Bing rankings.
When will Google support the shared token?
Google expects to support it in early 2027. Until then, Google-Extended remains Google’s active company-specific control.
What is the difference between AI training and grounding?
Training uses content to adjust a model’s parameters. Grounding retrieves a webpage when generating an answer, often providing a citation or link. A provider may allow publishers to restrict one use while permitting the other.
Do I still need Google-Extended?
Yes, for now. Google says company-specific controls will continue to work alongside the shared signal, allowing website owners to set a broad preference and then tailor their Google settings.
Will an opt-out remove content already used for training?
No. The preference applies to future training and does not work retroactively.
Does Disallow under User-agent: * count as an AI training opt-out?
No. Under the current draft, wildcard groups do not match the proposed policy token. An explicit __usage_ai-training group is required.
