Crawlers & control files
AI Bot Blocking
Using robots.txt and server rules to stop AI crawlers from accessing your site.
AI bot blocking is the practice of using robots.txt directives and server-level rules to stop specific AI crawlers from fetching your pages. Done precisely, it lets you keep training crawlers out while allowing the retrieval crawlers that earn citations. Done bluntly, it removes you from AI answers entirely.
The frequent mistake is a blanket disallow that catches the search and retrieval bots along with the training ones, which quietly erases a brand from ChatGPT, Perplexity and AI-Overview citations. Verify that your CDN or firewall rules agree with robots.txt, since an aggressive bot-management setting can block retrieval crawlers even when robots.txt allows them.
The recommended default for most B2B sites is to allow retrieval and search crawlers and make a deliberate, documented choice about training crawlers.
Related terms
AI Crawler
A bot that fetches pages to train or feed AI, not to build a search index.
GPTBot
OpenAI's crawler that gathers web content to train its models.
OAI-SearchBot
OpenAI's crawler that indexes pages so they can be cited in ChatGPT search.
PerplexityBot
Perplexity's crawler that surfaces and links websites in its answer engine.
Google-Extended
A robots.txt token that controls whether Google may train Gemini on your content.
Want to know where you stand in AI answers?
We’ll show you which engines cite you, and which don’t.
Get a free AI citation audit