Crawlers & control files
OAI-SearchBot
OpenAI's crawler that indexes pages so they can be cited in ChatGPT search.
OAI-SearchBot is OpenAI's retrieval crawler: it fetches and indexes pages so ChatGPT can surface and cite them in its search answers. It is the bot that matters for being cited in ChatGPT today, distinct from GPTBot, which gathers training data.
If you want ChatGPT to name and link your pages, OAI-SearchBot must be allowed to crawl them. Blocking it — directly, or by blocking everything in a blanket rule — is a common, self-inflicted reason a brand is absent from ChatGPT answers.
The clean policy is to allow OAI-SearchBot regardless of how you feel about training, so retrieval and citation stay open even if you restrict training crawlers.
Related terms
GPTBot
OpenAI's crawler that gathers web content to train its models.
AI Crawler
A bot that fetches pages to train or feed AI, not to build a search index.
PerplexityBot
Perplexity's crawler that surfaces and links websites in its answer engine.
AI Bot Blocking
Using robots.txt and server rules to stop AI crawlers from accessing your site.
Citation (AI)
When an AI answer attributes or links a specific page as its source.
Want to know where you stand in AI answers?
We’ll show you which engines cite you, and which don’t.
Get a free AI citation audit