What this checker tests
The scan runs the full CrawlWise audit. These checks answer this page's question; the report shows each one with its evidence.
- robots.txt is reachable and valid
discovery.robots - AI search and citation bots are allowed
discovery.ai-bots - AI crawlers get the same page as a regular fetch
discovery.ai-bot-access - Content usage preferences for AI are declared
discovery.content-signal - robots.txt allows CSS and JavaScript needed for rendering
discovery.resource-access
What the result tells you
The report lists every AI bot group your robots.txt names and the rule that applies to each one, then shows what each bot received when CrawlWise fetched the page as that bot: the status code, whether it got a challenge page, and how much of the text came back compared with a normal fetch.
Read the two parts together:
- Allowed in robots.txt, blocked at the edge. The most common problem. A CDN or bot protection rule rejects the bot before robots.txt matters. Check the AI crawler settings in Cloudflare or your firewall.
- Disallowed in robots.txt. Usually a named group, such as
User-agent: GPTBot, or aUser-agent: *group withDisallow: /. Remember that a named group replaces the*group for that bot. - Training blocked, search allowed. A deliberate policy many sites choose. CrawlWise passes it.
Which bots matter for AI search
Training crawlers (GPTBot, ClaudeBot, CCBot) decide whether your pages can be used to train
models. Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) build the index that
AI answers cite. User fetchers (ChatGPT-User, Claude-User, Perplexity-User) load a page
when someone asks an assistant about it. Blocking the second or third group removes you from AI
answers; blocking the first does not.
Fix a blocked crawler
- Allow the AI search and user bots in robots.txt. The robots.txt guide has a policy you can copy.
- If you use Cloudflare, review the AI crawler settings described in the Cloudflare guide.
- Let crawlers fetch the CSS and JavaScript the page needs, so a rendering crawler sees the same layout as a visitor.
- Run the check again. robots.txt changes reach the crawlers after they refetch the file, which can take up to a day.
FAQ
How do I check if GPTBot can crawl my site?
Paste your URL above. CrawlWise reads your robots.txt rules for GPTBot, OAI-SearchBot, ChatGPT-User and the other AI bots, and fetches the page with their user agents to see whether a CDN or firewall answers them differently.
Does blocking GPTBot remove my site from ChatGPT search?
No. GPTBot collects pages for training. ChatGPT search uses OAI-SearchBot, and ChatGPT-User fetches pages a person asks about. OpenAI treats each setting separately, so you can block training and stay in search.
My robots.txt allows AI bots. Why are they still blocked?
robots.txt is only a request. Cloudflare's AI crawler settings, bot protection and firewall rules answer before your server does, often with a 403, 402 or a challenge page. The bot never sees your robots.txt rules.
Can this checker prove what the real crawlers get?
Not completely. CrawlWise sends the bots' user agents from its own addresses, not from the crawlers' published IP ranges. A firewall that verifies crawler IPs can treat the real bot differently, so a mismatch for a search bot is a warning, not a failure.
Related guides
More free checkers
Last updated .