Free tools

Free · no account

AI crawler checker

Find out whether ChatGPT, Claude and Perplexity can reach your pages. The check reads your robots.txt for every major AI bot, then fetches the page as those bots to catch blocks at the CDN or firewall.

Add up to 2 more pages (optional)

Paste a link or type a domain · No account · Private report

Want to choose which parts of the audit open your report? Use the full audit form.

What this checker tests

The scan runs the full CrawlWise audit. These checks answer this page's question; the report shows each one with its evidence.

What the result tells you

The report lists every AI bot group your robots.txt names and the rule that applies to each one, then shows what each bot received when CrawlWise fetched the page as that bot: the status code, whether it got a challenge page, and how much of the text came back compared with a normal fetch.

Read the two parts together:

  • Allowed in robots.txt, blocked at the edge. The most common problem. A CDN or bot protection rule rejects the bot before robots.txt matters. Check the AI crawler settings in Cloudflare or your firewall.
  • Disallowed in robots.txt. Usually a named group, such as User-agent: GPTBot, or a User-agent: * group with Disallow: /. Remember that a named group replaces the * group for that bot.
  • Training blocked, search allowed. A deliberate policy many sites choose. CrawlWise passes it.

Training crawlers (GPTBot, ClaudeBot, CCBot) decide whether your pages can be used to train models. Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) build the index that AI answers cite. User fetchers (ChatGPT-User, Claude-User, Perplexity-User) load a page when someone asks an assistant about it. Blocking the second or third group removes you from AI answers; blocking the first does not.

Fix a blocked crawler

  1. Allow the AI search and user bots in robots.txt. The robots.txt guide has a policy you can copy.
  2. If you use Cloudflare, review the AI crawler settings described in the Cloudflare guide.
  3. Let crawlers fetch the CSS and JavaScript the page needs, so a rendering crawler sees the same layout as a visitor.
  4. Run the check again. robots.txt changes reach the crawlers after they refetch the file, which can take up to a day.

FAQ

How do I check if GPTBot can crawl my site?

Paste your URL above. CrawlWise reads your robots.txt rules for GPTBot, OAI-SearchBot, ChatGPT-User and the other AI bots, and fetches the page with their user agents to see whether a CDN or firewall answers them differently.

Does blocking GPTBot remove my site from ChatGPT search?

No. GPTBot collects pages for training. ChatGPT search uses OAI-SearchBot, and ChatGPT-User fetches pages a person asks about. OpenAI treats each setting separately, so you can block training and stay in search.

My robots.txt allows AI bots. Why are they still blocked?

robots.txt is only a request. Cloudflare's AI crawler settings, bot protection and firewall rules answer before your server does, often with a 403, 402 or a challenge page. The bot never sees your robots.txt rules.

Can this checker prove what the real crawlers get?

Not completely. CrawlWise sends the bots' user agents from its own addresses, not from the crawlers' published IP ranges. A firewall that verifies crawler IPs can treat the real bot differently, so a mismatch for a search bot is a warning, not a failure.

Related guides

More free checkers

Run the free check

Last updated .