Crawlers & control files
ClaudeBot
Anthropic's web crawler that collects public content for Claude.
ClaudeBot is Anthropic's web crawler, which collects public content used by Claude. As with other providers, Anthropic operates distinct crawlers for different purposes, so the robots.txt decision is again about separating training access from retrieval access.
For B2B teams whose buyers use Claude for research, keeping the retrieval path open is what preserves the chance of being surfaced and cited there. Blanket blocks remove that chance.
Treat ClaudeBot the way you treat the OpenAI and Perplexity bots: make a deliberate, documented choice per bot rather than a single catch-all rule.
Related terms
AI Crawler
A bot that fetches pages to train or feed AI, not to build a search index.
GPTBot
OpenAI's crawler that gathers web content to train its models.
PerplexityBot
Perplexity's crawler that surfaces and links websites in its answer engine.
AI Bot Blocking
Using robots.txt and server rules to stop AI crawlers from accessing your site.
Google-Extended
A robots.txt token that controls whether Google may train Gemini on your content.
Want to know where you stand in AI answers?
We’ll show you which engines cite you, and which don’t.
Get a free AI citation audit