How to Block AI Training but Stay in AI Overviews and AI Answers: The 2026 Guide to the New Opt-Out Signal

By Saj Ali · Last updated 31 Oct 2026

To block AI training but stay in Google’s AI Overviews, disallow Google-Extended (and, from early 2027, the shared __usage_ai-training token) in robots.txt, but keep Googlebot allowed and leave Search Console’s Search generative AI control set to “Include”. Google-Extended doesn’t affect inclusion in Google Search, and AI Overviews and AI Mode are part of Search (Google crawler docs; Search Console Help).

For ChatGPT, block GPTBot and allow OAI-SearchBot (OpenAI). For Claude, block ClaudeBot and allow Claude-SearchBot and Claude-User (Anthropic). For Bing and Copilot, the shared token covers training, and a separate page-level “AI grounding” tag decides whether your pages can inform AI answers (Microsoft Bing, 30 Oct 2026). This guide explains the new signal Google, Microsoft, OpenAI and Anthropic backed on 30 October 2026, the trade-offs, and a robots.txt you can adapt. If you want to see whether AI answers still cite you after a change, our round-up of the best AI visibility optimisation tools covers the trackers.

What changed on 30 October 2026?

Four companies published coordinated announcements on 30 October 2026:

• Google said it plans to support a shared “policy token” from the IETF’s AI Preferences (AIPREF) work. “We aim to support this new control along with other AI providers in 2027,” it wrote (Google Search Central, 30 Oct 2026). Its crawler docs say it intends to support the signal “in early 2027” (Google crawler docs).
• OpenAI said it will honour the common signal for “foundational generative AI training”, aims to implement it “in the coming months”, and will keep its existing crawler controls (OpenAI, 30 Oct 2026).
• Microsoft Bing adopted the token for AI training and added a separate AI grounding preference, with a transition timetable running into 2027 (Microsoft Bing).
• Anthropic: the IETF draft, draft-kelley-aipref-training-preference-signal-00, was written by K. Kelley of Anthropic and published on 30 October 2026 (IETF). PPC Land reports that Kelley told the IETF ai-control mailing list that Anthropic “has already started implementing support”, without giving a date (PPC Land, 31 Oct 2026).

Google linked the move to European law. It said the proposal is consistent with the EU Code of Practice for General Purpose AI model providers, which recognises robots.txt as a primary way for publishers to state machine-readable preferences (Google Search Central).

Training vs grounding: what is the difference?

These two words decide almost every choice in this guide, so it’s worth being precise:

Sources: IETF draft and AIPREF vocabulary, Google crawler docs, Search Console Help, Microsoft Bing. Checked 31 Oct 2026.

The IETF draft uses the AIPREF vocabulary’s definition of AI training: “Using an asset to modify the learned parameters of a generative AI model” (IETF draft). Retrieval at answer time doesn’t change a model’s parameters, which is why PPC Land reads grounding as falling outside the token (PPC Land).

For marketers, the rule of thumb is simple. Training is where your content helps build the model and you usually get nothing visible back. Grounding is where AI answers cite you, and it’s where AI search traffic comes from. Most brands want to keep grounding. Whether to allow training is a business and legal decision. Our generative engine optimisation guide explains why citations matter.

What is the __usage_ai-training token?

It’s a reserved user-agent name you put in robots.txt to state a training preference to every AI company that honours it, instead of naming each crawler. The draft’s example (IETF draft):

Key rules from the draft:

• It starts with two underscores. Some news coverage dropped them, but the draft, Google and Microsoft all use __usage_ai-training. Matching is case-insensitive.
• A User-Agent: * group must not be read as applying to the token. If you don’t add a __usage_ai-training group, you haven’t expressed a preference.
• Proprietary tokens such as Google-Extended don’t count as the shared signal, and the shared signal doesn’t replace them.
• It isn’t a crawl directive. It states how content may be used. Companies “MAY” choose not to fetch, but blocking a crawler is still done with that crawler’s own name.
• Path rules work, so you can opt most of the site out and leave a press kit or docs folder in.

Two more points PPC Land drew from the draft: private licensing deals “MAY override” the signal, and the preference that applies is the one in your robots.txt when the content was retrieved. The draft doesn’t address content collected before you add the token (PPC Land).

What does each company do with the signal?

Sources: Google Search Central, Google crawler docs, OpenAI, Microsoft Bing, Anthropic Help Center, PPC Land. Checked 31 Oct 2026.

Google: the shared token also removes Gemini app grounding

Google says: “When a website uses this policy token to opt out of training in its robots.txt file (and Google-Extended is not explicitly specified), Google plans to interpret that as if the publisher had disallowed Google-Extended” (Google Search Central). Google-Extended covers training of Gemini models and grounding in Gemini Apps and Grounding with Google Search on Vertex AI (Google crawler docs).

So at Google, the shared token switches off both. PPC Land’s reading is that Google offers no separate grounding token, so there’s no single Google setting for “cite me in Gemini but don’t train on me” (PPC Land). Google’s own wording is that sites that “specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps” (Google Search Central).

What it doesn’t touch: Google-Extended “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal” (Google crawler docs). AI Overviews and AI Mode are Search features, governed by Googlebot and the Search Console control.

OpenAI: separate bots for training and search

OpenAI’s documentation says each setting is independent: you can allow OAI-SearchBot “in order to appear in search results while disallowing GPTBot”. Sites that block OAI-SearchBot “will not be shown in ChatGPT search answers, though can still appear as navigational links”. Changes can take about 24 hours to take effect for search (OpenAI crawler docs).

Anthropic: three bots

Anthropic says restricting ClaudeBot signals that future content should be excluded from training. Blocking Claude-User stops Claude fetching your pages for users’ questions, and blocking Claude-SearchBot stops indexing for search. Both of those “may reduce your site’s visibility”. Rules need adding for every subdomain (Anthropic Help Center).

How to block AI training but stay in AI Overviews

Here is the step-by-step version for Google, then the other assistants.

1. Keep Googlebot allowed. AI Overviews and AI Mode draw on the Search index. Blocking Googlebot removes you from Search entirely.
2. Leave the Search Console control on “Include”. In Search Console, go to Settings > Search generative AI. “Include my site’s links and content in Search generative AI features” is the default. Choosing “Exclude” removes your links and content from AI Overviews, AI Mode and generative AI features in Discover. Google says exclusion generally takes effect within a few days (Search Console Help).
3. Don’t add nosnippet to pages you want cited. Google says nosnippet, data-nosnippet, max-snippet and noindex limit what appears from your pages in Search, including AI features (Google AI features docs).
4. Disallow Google-Extended. Add User-agent: Google-Extended and Disallow: /. Search Console Help says Google-Extended is the way “to limit training of the models used to generate responses in Search generative AI features”, and it doesn’t affect Search inclusion (Search Console Help; Google crawler docs).
5. Accept the trade-off. This also removes you from grounding in the Gemini app. If Gemini app citations matter more than training, allow Google-Extended instead.
6. Add the shared token (__usage_ai-training) for other companies. Google only maps the shared token to Google-Extended when Google-Extended “is not explicitly specified”, so a named Google-Extended group still decides what Google does (Google Search Central).
7. For ChatGPT, disallow GPTBot and allow OAI-SearchBot.
8. For Claude, disallow ClaudeBot and allow Claude-SearchBot and Claude-User.
9. Test. Use Search Console’s URL Inspection to check what Googlebot sees, then watch AI citations for a few weeks.

What is AI grounding in Bing, and how do you control it?

Microsoft defines AI grounding as a page-level preference for whether your content is used “to help inform generated responses”. It has three settings (Microsoft Bing):

Source: Microsoft Bing blog, 30 Oct 2026.

There is no site-wide robots.txt setting for grounding. Microsoft warns that DisallowExtended or Disallow “will significantly reduce your content’s visibility in AI-generated experiences”, without affecting normal Bing results (Microsoft Bing).

For training, Microsoft reads the shared robots.txt token and adds page-level overrides: <meta name="Bingbot" content="ai-training:disallow"> or X-Robots-Tag: Bingbot: ai-training:disallow. A page-level training setting overrides robots.txt (Microsoft Bing).

Two limits to know. These settings govern use, not crawl access, so an Allow doesn’t override a robots.txt rule that blocks Bingbot. And Microsoft has to recrawl a page to see a new setting. IndexNow can make a recrawl more likely but doesn’t guarantee a date. Microsoft is clear that “Neither choice guarantees inclusion, traffic, or payment” (Microsoft Bing).

If you want to block training but stay in Copilot answers, use the shared token (or the page-level ai-training tag) and leave grounding on Allow. If you only object to Microsoft reselling grounding to other companies’ AI, DisallowExtended is the middle option.

A robots.txt you can copy

This file blocks AI training at Google, OpenAI, Anthropic and Microsoft (once each supports the shared token), while keeping Google Search, AI Overviews, AI Mode, ChatGPT search, Claude search and Bing AI answers open. Adapt it to your own crawl rules and test before you deploy.

A few notes. If you want Gemini app citations, change the Google-Extended group to Allow: /, which also allows Gemini training. ChatGPT-User isn’t listed because OpenAI says robots.txt rules “may not apply” to user-initiated visits and that it isn’t used to decide what appears in search (OpenAI crawler docs). Bing grounding is set per page in your HTML or headers, not here.

If you run WordPress, robots.txt is usually managed by your SEO plugin. Our SEO tools category and AI SEO tools category list options, and Claude SEO (see our Claude SEO review) can audit technical set-ups like this.

Key dates for 2026 and 2027

If your site uses NoArchive or NoCache to keep content out of Copilot, act before 1 March 2027. Microsoft says those sites should publish the new page-level grounding preference and record a temporary transition preference in Bing Webmaster Tools by then. After 1 March 2027, NoArchive and NoCache “will no longer be read as a preference for training or grounding”, and both new preferences default to Allow when none is published (Microsoft Bing). PPC Land also points out that once Microsoft detects the new preferences on a site, it disregards any NoArchive and NoCache left there (PPC Land).

Mistakes to avoid

• Blocking the wrong bot. Blocking OAI-SearchBot or Claude-SearchBot removes you from those assistants’ search answers. Block the training bots (GPTBot, ClaudeBot) instead.
• Assuming the shared token covers everything. It covers training only. At Microsoft, grounding has its own tag. At Google, it maps to Google-Extended, which also covers Gemini app grounding.
• Relying on User-agent: *. The draft says a wildcard group must not be read as matching the token. Write the token out.
• Typing one underscore. The token is __usage_ai-training, with two.
• Using Search Console’s Exclude to stop training. It doesn’t affect training, and it removes you from AI Overviews and AI Mode. Use Google-Extended for training.
• Expecting it to be retroactive. The preference applies from when content is retrieved. It doesn’t pull content out of models already trained.
• Forgetting subdomains. Every subdomain has its own robots.txt.

Once your rules are live, track the result. AI visibility platforms such as Peec AI and Profound show whether ChatGPT, Gemini and Copilot still cite you. For strategy, see our answer engine optimisation guide, and if you’d rather hand it over, our directory lists AI SEO and GEO agencies in the UK. Our GEO tools category has more trackers, our Peec AI listing has its plans, and our guides to ChatGPT for SEO and the best LLM for SEO cover how each assistant finds content.

FAQs

How do I block AI training but stay in AI Overviews? Disallow Google-Extended in robots.txt, keep Googlebot allowed, and leave Search Console’s Search generative AI control set to Include. Google says Google-Extended doesn’t affect inclusion in Google Search, and AI Overviews and AI Mode are Search features. Note that blocking Google-Extended also removes you from grounding in the Gemini app.

What is the __usage_ai-training token? It’s a shared robots.txt user-agent name from an IETF draft published on 30 October 2026. A Disallow under it tells every AI company that honours it not to use your content for AI training. Google plans to support it in early 2027, Microsoft from 1 February 2027, OpenAI in the coming months, and Anthropic has started implementing it.

What is AI grounding in Bing? AI grounding is when Microsoft uses your page to help inform an AI-generated answer at the time someone asks, in Bing, Copilot or Microsoft 365, usually with a citation. You control it per page with a Bingbot ai-grounding meta tag or X-Robots-Tag header set to disallow or disallowextended. There is no site-wide robots.txt setting for it.

Does blocking GPTBot remove my site from ChatGPT? No. GPTBot is OpenAI’s training crawler. OpenAI uses OAI-SearchBot to surface sites in ChatGPT search, and says you can allow OAI-SearchBot while disallowing GPTBot.

Will the shared token remove me from Google Search or AI Overviews? No. Google will treat it as a Google-Extended disallow, and Google says Google-Extended doesn’t affect inclusion in Google Search and isn’t a ranking signal. To leave AI Overviews and AI Mode you would use Search Console’s Search generative AI control.

Does User-agent: * cover the new AI training token? No. The IETF draft says a wildcard group must not be read as matching the token. If you don’t add a group for __usage_ai-training, you haven’t expressed a training preference.

Does opting out remove my content from models already trained? No. The draft says the preference that applies is the one in your robots.txt when the content was retrieved, and it doesn’t address content collected before you add the token.

Sources

Every figure above comes from the sources below, opened and checked on 31 October 2026. Vendor and third-party figures are theirs, not ours.

1. Google Search Central: AI training control (30 Oct 2026)
2. Google: Common crawlers, Google-Extended
3. Search Console Help: Search generative AI control
4. Google Search Central: AI features and your website
5. OpenAI: Helping websites express AI training preferences (30 Oct 2026)
6. OpenAI: Overview of OpenAI crawlers
7. Microsoft Bing: Clearer preferences for training and grounding (30 Oct 2026)
8. IETF: draft-kelley-aipref-training-preference-signal-00
9. Anthropic: Does Anthropic crawl data from the web?
10. PPC Land: Google, Microsoft, OpenAI and Anthropic back one AI training opt-out (31 Oct 2026)
11. GOV.UK: Google search publisher conduct requirement

Last updated 31 Oct 2026. Digital Marketing Supermarket is an independent directory. Nobody paid for this article, and no company named in it saw it before publication.