Crawlers & control files
GPTBot
OpenAI's crawler that gathers web content to train its models.
GPTBot is OpenAI's crawler for gathering public web content to train its models. It is a training crawler, not the mechanism that lets ChatGPT cite a live page — that is OAI-SearchBot. Blocking GPTBot in robots.txt keeps your content out of future training data but does not remove you from ChatGPT's search citations.
This difference trips up a lot of teams. A blanket 'block GPTBot' is a defensible stance on training, but it is often mistaken for 'block ChatGPT', which it is not. If your goal is to be cited in ChatGPT answers, the crawler you must keep allowed is OAI-SearchBot.
The practical setup is a deliberate robots.txt: decide your stance on training separately from your stance on retrieval, and label each named bot accordingly.
Related terms
AI Crawler
A bot that fetches pages to train or feed AI, not to build a search index.
OAI-SearchBot
OpenAI's crawler that indexes pages so they can be cited in ChatGPT search.
Google-Extended
A robots.txt token that controls whether Google may train Gemini on your content.
CCBot (Common Crawl)
Common Crawl's web crawler, whose public dataset trains many language models.
AI Bot Blocking
Using robots.txt and server rules to stop AI crawlers from accessing your site.
Want to know where you stand in AI answers?
We’ll show you which engines cite you, and which don’t.
Get a free AI citation audit