Don't Block AI Crawlers: Why Your Bot Protection Is Hurting Your AI Visibility

Here's a failure mode we see constantly: a company invests in content, wins great reviews, builds a solid product — and AI assistants still describe them vaguely, wrongly, or not at all.

The cause is often embarrassingly simple: their website blocks the crawlers AI platforms use to read it.

If ChatGPT, Perplexity, or Claude can't fetch your pages, they can't learn what you actually do. They'll either skip you when buyers ask for recommendations, or worse — guess based on your company name alone. We've seen an automotive vinyl-wrap company described as an "LED lighting" brand purely because the AI couldn't read the site and had nothing else to go on.

This guide explains how AI crawlers work, how to check whether you're blocking them, and how to allow the trusted ones without opening the door to abusive bots.

---

How AI Platforms Read Your Website

AI platforms access your content in three distinct ways, each with its own crawler:

1. Training crawlers collect content to train future models. Examples: GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google's AI training opt-out token), CCBot (Common Crawl, used by many labs).

2. Search/index crawlers build the retrieval indexes AI answers draw from. Examples: OAI-SearchBot (powers ChatGPT search results), PerplexityBot (builds Perplexity's index).

3. On-demand user fetchers retrieve a page live because a user's question triggered it. Examples: ChatGPT-User (ChatGPT browsing on behalf of a user), Perplexity-User. Blocking these means the AI literally cannot cite you in the moment a buyer is asking.

Blocking category 1 is a legitimate business choice — some publishers don't want to contribute training data. But blocking categories 2 and 3 directly removes you from AI answers. Most bot-protection setups don't distinguish between them: they block everything that isn't a human browser.

---

The Two Ways You Might Be Blocking AI Crawlers

1. robots.txt rules

The explicit way. If your robots.txt contains:

User-agent: GPTBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

…then well-behaved AI crawlers will honor it and never read your site. Some CMS plugins and "privacy" presets add these rules by default — many site owners don't even know they're there.

Check yours right now: open yourdomain.com/robots.txt and search for GPTBot, ChatGPT-User, OAI-SearchBot, PerplexityBot, ClaudeBot, and Google-Extended.

2. WAF / bot protection (the silent killer)

Cloudflare, Akamai, AWS WAF, and similar services score every request and challenge or block anything that looks automated. Default and "under attack" configurations frequently return 403 Forbidden to AI crawlers — no robots.txt rule required, and nothing visible to you as a human visitor.

This one is sneaky because your site looks fine in your browser while being a brick wall to every AI platform. The only way to notice is to test with a crawler user-agent or check your WAF logs for blocked bot traffic.

---

How to Allow Trusted AI Crawlers

Fix robots.txt

Explicitly allow the crawlers you want:

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: ClaudeBot
Allow: /

If you want to opt out of model training but stay visible in AI answers, block only the training crawlers (GPTBot, Google-Extended, CCBot) and allow the search and user-fetch crawlers listed above.

Fix your WAF

• Cloudflare: Security → Bots → enable "Verified Bots" allowance. OpenAI, Perplexity, and Anthropic crawlers are verified — Cloudflare confirms them by published IP ranges, so spoofers don't get through. If you use custom firewall rules, add a skip rule for cf.verified_bot_category eq "AI Crawler" (or the specific verified bots you want).
• AWS WAF / Akamai / others: allowlist the official published IP ranges — OpenAI, Anthropic, and Perplexity each publish theirs — rather than matching on user-agent strings alone (user-agents can be spoofed; IPs can't).
• Rate limiting: AI crawlers respect Crawl-delay and back off on 429s. Prefer rate limiting over hard 403 blocks if load is your concern.

Verify the fix

After changing settings, fetch your homepage with a crawler user-agent:

curl -I -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot" https://yourdomain.com/

A 200 means you're readable. A 403 means the wall is still up.

---

Why This Matters More Every Month

A growing share of buying research starts in an AI assistant instead of a search box. When someone asks "best paint protection film in Australia" and the AI can't read your site, one of two things happens:

1. You're invisible — competitors with readable sites fill the answer.
2. You're misrepresented — the AI reconstructs you from stale third-party mentions or, with nothing to go on, guesses from your name.

Both are worse than any scraping downside for a commercial brand. You spent years building the site; make sure the systems buyers actually ask can read it.