Crawlers & control files
Google-Extended
A robots.txt token that controls whether Google may train Gemini on your content.
Google-Extended is a robots.txt token that controls whether Google may use your content to train and ground its Gemini models. Crucially, it does not affect whether your pages appear in Google Search or in AI Overviews — those are fed by Googlebot, which is governed separately. Disallowing Google-Extended limits Gemini's use of your content without touching your search visibility.
This separation lets you opt out of Gemini training while staying fully present in Google Search and its AI Overviews. Teams that want to limit training exposure can disallow Google-Extended without harming their search or AI-Overview visibility.
As always, the key is not to conflate the training token with search access — blocking Google-Extended is safe for visibility; blocking Googlebot is not.
Related terms
GPTBot
OpenAI's crawler that gathers web content to train its models.
AI Crawler
A bot that fetches pages to train or feed AI, not to build a search index.
AI Bot Blocking
Using robots.txt and server rules to stop AI crawlers from accessing your site.
AI Overviews
Google's AI-generated answer box at the top of search results.
CCBot (Common Crawl)
Common Crawl's web crawler, whose public dataset trains many language models.
Want to know where you stand in AI answers?
We’ll show you which engines cite you, and which don’t.
Get a free AI citation audit