Sixteen named AI crawlers — GPTBot, ClaudeBot, PerplexityBot and 13 more — must be able to fetch your pages before any content optimization matters, and on many of the sites we audit at least one of them is blocked by accident. Crawler access is a yes/no gate, not a ranking factor.
The crawlers we check on every audit
Our standard audit verifies access for 16 user agents across the major AI ecosystems:
- OpenAI: GPTBot (training), OAI-SearchBot (search index), ChatGPT-User (live browsing when a user asks)
- Anthropic: ClaudeBot, Claude-Web, anthropic-ai
- Perplexity: PerplexityBot, Perplexity-User
- Google: Google-Extended (controls AI training use), plus Applebot / Applebot-Extended on the Apple side
- Others: Bytespider (ByteDance), Amazonbot, meta-externalagent (Meta), cohere-ai, YouBot
Three of these deserve special attention: the "-User" agents (ChatGPT-User, Perplexity-User) fetch your page at the moment a user asks a question. Blocking them doesn't just affect some future training run — it removes you from live answers today.
How sites block AI crawlers by accident
- A blanket "Disallow: /" for unknown bots. Common in robots.txt templates that allowlist only Googlebot and Bingbot. Every AI crawler falls into the "unknown" bucket.
- WAF / bot-protection defaults. Cloudflare and similar services ship AI-bot blocking rules that can be on without the site owner ever choosing them. The robots.txt looks fine; the crawler still gets a 403.
- Rate-limit rules tuned for humans. AI crawlers get throttled into effective invisibility.
The five-minute check
- Open "yourdomain.com/robots.txt" and read it as a machine would: does any "User-agent" group above match (or fall through to) each crawler name, and is it allowed?
- Fetch your homepage impersonating a crawler and check the status code. From any terminal: curl -sI -A "GPTBot" https://yourdomain.com/ — then read the first line of the response. Anything other than "200" (a "403", "406", "429") means a layer above robots.txt is blocking — usually the WAF.
- Confirm your robots.txt declares your sitemap ("Sitemap: https://yourdomain.com/sitemap.xml") and that the sitemap URL actually returns valid XML.
Should you allow all of them?
That's a business decision, and there are legitimate reasons to say no to training crawlers specifically (Google-Extended, GPTBot) while allowing search/browsing agents. Our default recommendation for businesses that want AI visibility is to allow all 16 explicitly — an explicit "Allow" block per agent — so the intent is unambiguous and survives future template edits. But it's your content; make the choice deliberately rather than inheriting it from a template written before AI crawlers existed.
Access is the cheapest fix in all of GEO: a few lines in one text file. It's also the one that makes every other fix possible.