Crawlers & control files
AI Crawler
A bot that fetches pages to train or feed AI, not to build a search index.
An AI crawler is a bot that fetches web pages to train AI models or to feed live AI answers, rather than to build a traditional search index. The category splits in two: training crawlers that gather content to teach a model, and retrieval or search crawlers that fetch pages so a model can cite them in real time.
The distinction is the whole game for control. Blocking a training crawler keeps your content out of the next model version but does not affect whether you are cited today; blocking a retrieval crawler removes you from live AI answers right now. Teams that block indiscriminately often cut themselves out of citations they wanted to keep.
Knowing which named bot does which job — GPTBot and Google-Extended train, OAI-SearchBot and PerplexityBot retrieve — is what lets you allow the crawlers that earn citations while limiting the ones that only take training data.
Frequently asked
Should I block AI crawlers?
Usually not all of them. Blocking training-only bots is a reasonable choice, but blocking the retrieval and search crawlers removes you from live AI answers and the citations that come with them.
Related terms
GPTBot
OpenAI's crawler that gathers web content to train its models.
OAI-SearchBot
OpenAI's crawler that indexes pages so they can be cited in ChatGPT search.
ClaudeBot
Anthropic's web crawler that collects public content for Claude.
PerplexityBot
Perplexity's crawler that surfaces and links websites in its answer engine.
AI Bot Blocking
Using robots.txt and server rules to stop AI crawlers from accessing your site.
Want to know where you stand in AI answers?
We’ll show you which engines cite you, and which don’t.
Get a free AI citation audit