# AI crawler checker

Find out whether ChatGPT, Claude and Perplexity can reach your pages. The check reads your robots.txt for every major AI bot, then fetches the page as those bots to catch blocks at the CDN or firewall.

Run it at https://crawlwise.dev/tools/ai-crawler-checker: paste a public URL, no account needed. The scan runs the full CrawlWise audit and returns a private report.

## What this checker tests

- robots.txt is reachable and valid (`discovery.robots`)
- AI search and citation bots are allowed (`discovery.ai-bots`)
- AI crawlers get the same page as a regular fetch (`discovery.ai-bot-access`)
- Content usage preferences for AI are declared (`discovery.content-signal`)
- robots.txt allows CSS and JavaScript needed for rendering (`discovery.resource-access`)

## What the result tells you

The report lists every AI bot group your robots.txt names and the rule that applies to each one,
then shows what each bot received when CrawlWise fetched the page as that bot: the status code,
whether it got a challenge page, and how much of the text came back compared with a normal fetch.

Read the two parts together:

- **Allowed in robots.txt, blocked at the edge.** The most common problem. A CDN or bot
  protection rule rejects the bot before robots.txt matters. Check the AI crawler settings in
  Cloudflare or your firewall.
- **Disallowed in robots.txt.** Usually a named group, such as `User-agent: GPTBot`, or a
  `User-agent: *` group with `Disallow: /`. Remember that a named group replaces the `*` group
  for that bot.
- **Training blocked, search allowed.** A deliberate policy many sites choose. CrawlWise passes it.

## Which bots matter for AI search

Training crawlers (`GPTBot`, `ClaudeBot`, `CCBot`) decide whether your pages can be used to train
models. Search crawlers (`OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot`) build the index that
AI answers cite. User fetchers (`ChatGPT-User`, `Claude-User`, `Perplexity-User`) load a page
when someone asks an assistant about it. Blocking the second or third group removes you from AI
answers; blocking the first does not.

## Fix a blocked crawler

1. Allow the AI search and user bots in robots.txt. The
   [robots.txt guide](/guides/ai-crawlers-robots-txt) has a policy you can copy.
2. If you use Cloudflare, review the AI crawler settings described in the
   [Cloudflare guide](/guides/cloudflare-ai-crawlers).
3. Let crawlers fetch the CSS and JavaScript the page needs, so a rendering crawler sees the
   same layout as a visitor.
4. Run the check again. robots.txt changes reach the crawlers after they refetch the file, which
   can take up to a day.

## FAQ

### How do I check if GPTBot can crawl my site?

Paste your URL above. CrawlWise reads your robots.txt rules for GPTBot, OAI-SearchBot, ChatGPT-User and the other AI bots, and fetches the page with their user agents to see whether a CDN or firewall answers them differently.

### Does blocking GPTBot remove my site from ChatGPT search?

No. GPTBot collects pages for training. ChatGPT search uses OAI-SearchBot, and ChatGPT-User fetches pages a person asks about. OpenAI treats each setting separately, so you can block training and stay in search.

### My robots.txt allows AI bots. Why are they still blocked?

robots.txt is only a request. Cloudflare's AI crawler settings, bot protection and firewall rules answer before your server does, often with a 403, 402 or a challenge page. The bot never sees your robots.txt rules.

### Can this checker prove what the real crawlers get?

Not completely. CrawlWise sends the bots' user agents from its own addresses, not from the crawlers' published IP ranges. A firewall that verifies crawler IPs can treat the real bot differently, so a mismatch for a search bot is a warning, not a failure.

## Related guides

- [AI crawlers in robots.txt: what to allow and block](https://crawlwise.dev/guides/ai-crawlers-robots-txt.md)
- [Is Cloudflare blocking AI crawlers on your site?](https://crawlwise.dev/guides/cloudflare-ai-crawlers.md)

---

Last updated 14 November 2026.

HTML version: <https://crawlwise.dev/tools/ai-crawler-checker>

Every CrawlWise page in Markdown: <https://crawlwise.dev/llms.txt>
