Crawlers & control files
CCBot (Common Crawl)
Common Crawl's web crawler, whose public dataset trains many language models.
CCBot is the crawler operated by Common Crawl, a non-profit that publishes a large, free dataset of web pages. That dataset has been a major training source for many large language models, so CCBot's reach extends well beyond any single AI company.
Blocking CCBot limits how much of your content flows into the many models trained on Common Crawl data, but like other training crawlers it has no bearing on live retrieval or citation. It is a training-exposure lever, not a visibility one.
Because so many models draw on Common Crawl, a block here is broad — which is a reason some teams allow it (to maximise presence in model knowledge) and others restrict it (to limit uncompensated training use).
Related terms
GPTBot
OpenAI's crawler that gathers web content to train its models.
Google-Extended
A robots.txt token that controls whether Google may train Gemini on your content.
AI Crawler
A bot that fetches pages to train or feed AI, not to build a search index.
AI Bot Blocking
Using robots.txt and server rules to stop AI crawlers from accessing your site.
Machine-Readable Content
Content encoded so software can parse its meaning, not just display its text.
Want to know where you stand in AI answers?
We’ll show you which engines cite you, and which don’t.
Get a free AI citation audit