Confirm robots.txt allows AI answer-engine crawlers

AI answer engines like ChatGPT, Claude, and Perplexity crawl the web with their own bots, separate entirely from Googlebot. If your robots.txt blocks them — even by accident, through a blanket rule meant for something else — you disappear from AI-generated answers completely, a fast-growing source of traffic that traditional search-only SEO misses entirely.

AI answer engines like ChatGPT, Claude, and Perplexity crawl the web with their own bots, separate from Googlebot. If your robots.txt blocks them — even accidentally, by blanket-blocking all bots — you disappear from AI-generated answers entirely, a fast-growing source of traffic search-only SEO misses.

How to fix it

1. 1
   Check your current robots.txt for AI bot rules

   Look specifically for User-agent lines matching GPTBot, ClaudeBot, PerplexityBot, Google-Extended, or CCBot.
2. 2
   Decide deliberately, not accidentally

   Blocking these is a legitimate choice if you specifically want to opt out of AI training/citation — the problem is only when it happens by accident via a blanket rule.
3. 3
   Remove accidental blocks

   If a general "User-agent: *, Disallow: /" rule is unintentionally catching these bots too, add explicit Allow rules for the ones you want to permit.
4. 4
   Re-check periodically as new AI crawlers emerge

   This is a fast-evolving space — new bots from new AI products appear regularly, worth revisiting occasionally.