Should You Block or Allow AI Crawlers? A B2B Guide to GPTBot, ClaudeBot, PerplexityBot and Google-Extended
Sites that unblocked GPTBot, PerplexityBot, and ClaudeBot saw +186% AI-attributed traffic in 90 days. Here's the B2B robots.txt strategy for training vs search crawlers in 2026.
Quick Answer
Block or allow AI crawlers? For most B2B companies, allow the search and retrieval crawlers (OAI-SearchBot, PerplexityBot, ClaudeBot) because those are what earn you citations in ChatGPT, Perplexity, and Claude answers. Sites that unblocked GPTBot, PerplexityBot, and ClaudeBot in Q4 2025 saw +186% AI-attributed traffic within 90 days (Anagram, 2026). If you want to stay out of model training while remaining citable, block training-only bots (GPTBot, Google-Extended, CCBot, anthropic-ai) and allow the retrieval agents. Blocking everything is a visibility mistake most B2B teams cannot afford.
Here is the trap I see B2B teams fall into. Sometime in 2024, someone read a headline about AI companies scraping the web, added a blanket block to robots.txt, and moved on. Two years later they wonder why they never show up in ChatGPT or Perplexity.
They blocked their own visibility. And they usually did not mean to.
The confusion is understandable, because "AI crawler" covers two completely different jobs. Some bots crawl your site to train future models. Others crawl it in real time to answer a user's question with a citation to you. Block the first kind and you protect your content from training. Block the second kind and you erase yourself from the answers your buyers now read first. Most blanket blocks kill both.
This guide sorts out which bot does what, shows the exact robots.txt setups for the three common strategies, and covers the enforcement gotchas that trip people up in 2026.
Training Crawlers vs Search Crawlers
This one distinction drives every decision. Learn it and the rest is mechanical.
Training crawlers fetch your content to build or improve foundation models. Blocking them keeps your pages out of future training data. It does not directly cost you citations today. Examples: GPTBot (OpenAI training), Google-Extended (Google model training), CCBot (Common Crawl), anthropic-ai.
Search and retrieval crawlers fetch your content in real time so the assistant can answer a live question and cite a source. Blocking these removes you from the answer. Examples: OAI-SearchBot and ChatGPT-User (OpenAI search), PerplexityBot (Perplexity retrieval), ClaudeBot and Claude-Web (Anthropic).
Here is the mental model: training crawlers decide whether a model remembers you. Search crawlers decide whether an assistant can cite you right now. For AI visibility, the second group is the one you protect.
| Crawler | Operator | Job | Block to stay out of training | Allow to stay citable |
|---|---|---|---|---|
| GPTBot | OpenAI | Training | Block if desired | Not required for search |
| OAI-SearchBot | OpenAI | Search | Allow | |
| ChatGPT-User | OpenAI | Live user fetch | Allow | |
| Google-Extended | Training | Block if desired | Not required for search | |
| PerplexityBot | Perplexity | Retrieval | Allow | |
| ClaudeBot | Anthropic | Retrieval/training | Allow | |
| CCBot | Common Crawl | Training dataset | Block if desired | Not required |
A key point on Google. Google-Extended controls training use, but it does not control Googlebot, which is what powers Google Search and AI Overviews. Blocking Google-Extended does not remove you from AI Overviews. Blocking Googlebot would, and you almost never want that.
The Three Strategies
Strategy 1: Allow everything (maximum visibility)
This is the right default for most B2B companies whose goal is to be found and cited. You want your content in front of every engine your buyers use.
# Welcome mat for AI engines
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Google-Extended
Allow: /
The upside is concrete. Anagram reported that sites unblocking GPTBot, PerplexityBot, and ClaudeBot in Q4 2025 saw a +186% lift in AI-attributed traffic within 90 days. If your content is a competitive asset you want cited, this is the posture.
Strategy 2: Block training, allow search (the balanced play)
Use this if you have a real reason to keep content out of model training (proprietary research, licensed data, legal caution) but you still want to earn citations. You block the training-only agents and explicitly allow the retrieval ones.
# Keep content out of training
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: anthropic-ai
Disallow: /
# Stay citable in live answers
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: ClaudeBot
Allow: /
This is the setup I recommend for companies whose original research is their moat. You protect the asset from training while staying eligible for the citations that drive pipeline. It is more nuanced than most agencies bother to explain.
Strategy 3: Block everything (rarely right for B2B)
Full block makes sense for a narrow set: sites under strict licensing terms, paywalled content, or regulated data that legally cannot be exposed. For a normal B2B company trying to grow, this is self-sabotage. You are opting out of the fastest-growing discovery channel your buyers use.
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
If you are considering this, make sure the legal or licensing reason is real, not just leftover caution from a 2024 headline.
The Enforcement Gotchas Nobody Warns You About
robots.txt is a set of instructions, not a wall. Two 2026 realities matter.
Your CDN and robots.txt must agree. Research on Cloudflare's network found a meaningful share of sites accidentally blocking major AI crawlers at the CDN or WAF level while their robots.txt said "allow." The two layers were fighting each other, and the CDN won. If you allow a bot in robots.txt but your CDN bot-management rules block it, you are still invisible. Check both.
Some crawlers do not fully honor robots.txt. Cloudflare published evidence in August 2025 that Perplexity used undeclared crawlers rotating user agents, IPs, and ASNs to fetch content that robots.txt tried to block. The lesson cuts both ways: if you want to block a determined crawler, robots.txt alone may not be enough and you will need WAF-level rules. If you want to allow crawlers, make sure nothing at the edge is silently blocking them.
The practical move: after you set robots.txt, verify actual crawler access in your server or CDN logs. Do not assume the file did what you intended. This is exactly the kind of thing our technical AI visibility audit checks, because a misconfigured edge rule quietly undoes an otherwise perfect setup.
What to Do This Week
- Pull your current robots.txt and read it with the training-versus-search distinction in mind.
- If you have a blanket AI block and no licensing reason for it, move to Strategy 1 or 2.
- Explicitly allow OAI-SearchBot, ChatGPT-User, PerplexityBot, and ClaudeBot so you stay citable.
- Confirm your CDN or WAF is not blocking what robots.txt allows.
- Check your logs a week later to verify the crawlers are actually getting through.
Crawler access is the Structure-stage plumbing in the ANSWER Framework. It will not earn you citations on its own, but getting it wrong quietly caps everything else you do. Fix the plumbing first, then build the content and authority that actually get cited.
Frequently Asked Questions
Will blocking GPTBot remove me from ChatGPT answers? Not by itself. GPTBot is OpenAI's training crawler. ChatGPT's live answers rely on OAI-SearchBot and ChatGPT-User. To stay citable in ChatGPT while opting out of training, block GPTBot but allow OAI-SearchBot and ChatGPT-User.
Does blocking Google-Extended remove me from AI Overviews? No. Google-Extended controls training use. AI Overviews and Google Search run on Googlebot. Blocking Google-Extended does not affect your AI Overviews eligibility, but blocking Googlebot would.
Should a B2B company block AI crawlers? Usually no. For most B2B companies the goal is citations and pipeline, so you want the search and retrieval crawlers allowed. Block training-only bots only if you have a genuine reason to keep content out of model training.
Does robots.txt actually stop AI crawlers? It stops well-behaved ones. Cloudflare documented crawlers that evaded robots.txt by rotating user agents and IPs, so strict blocking requires WAF or CDN enforcement, not just a text file.
What is the difference between GPTBot and OAI-SearchBot? GPTBot crawls for OpenAI model training. OAI-SearchBot crawls to power ChatGPT's search and citations. Allowing OAI-SearchBot keeps you eligible to be cited; allowing GPTBot feeds training.
The Takeaway
For most B2B companies, allowing the search and retrieval crawlers is the visibility decision, not a risk. Block training bots if you have a real reason, keep the retrieval bots allowed either way, and make sure your CDN is not quietly overriding your robots.txt. The teams that unblocked these crawlers saw real traffic gains. The ones running a leftover 2024 blanket block are paying for it in citations they never earn.
Not sure whether AI crawlers can even reach your content right now? Run a free AI visibility audit and we will flag crawler and edge-level blocks along with your citation baseline.
Sources
- Anagram, "AI Crawlers Explained: GPTBot, ClaudeBot, PerplexityBot and How to Let Them In (2026)" (+186% AI-attributed traffic figure)
- Cloudflare, network analysis of accidental AI-crawler blocking at the CDN level (2026)
- Cloudflare, evidence of Perplexity undeclared-crawler activity (August 2025)
- DataImpulse, Pixis, and Mersel AI 2026 robots.txt and AI-crawler management guides
- OpenAI, Google, and Anthropic public crawler documentation (user-agent behavior)
Related Reading
Get AEO Insights Weekly
Join 500+ B2B marketers getting AI visibility tactics every Tuesday.
Ready to Get Your Brand Cited by AI?
See how your competitors show up in ChatGPT, Perplexity, and Gemini — and what it would take to get recommended.
