What this checker tests
The scan runs the full CrawlWise audit. These checks answer this page's question; the report shows each one with its evidence.
- robots.txt is reachable and valid
discovery.robots - AI search and citation bots are allowed
discovery.ai-bots - robots.txt allows CSS and JavaScript needed for rendering
discovery.resource-access
What the report shows
CrawlWise fetches /robots.txt from the scanned host and reports:
- the HTTP status and content type. A
401,403or429is reported as blocked, not as a failure, because the scanner could not read the file. A server error or an HTML page served at/robots.txtfails; - how many user-agent groups and rules the file has, and whether a rule disallows every path for all crawlers;
- which search and citation bots (
OAI-SearchBot,Claude-SearchBot,PerplexityBot,Googlebot,Bingbot) and which training bots (GPTBot,ClaudeBot,Google-Extended,CCBotand others) may fetch the site root; - whether Googlebot may fetch the page’s same-origin stylesheets and scripts, which Google needs to render the page.
Common mistakes
- An HTML page at /robots.txt. A single-page app that answers every path with its shell gives crawlers a file without rules.
Disallow: /left over from staging. One line blocks the whole site for every crawler that reads the*group.- Blocking
/assets/or/_next/. Crawlers can then fetch the HTML but not the CSS and JavaScript that render it. - Blocking every AI bot at once. A rule for
GPTBotdoes not coverOAI-SearchBot. Decide on search and training bots separately.
Fix it
Serve /robots.txt as text/plain with a Sitemap: line and only the rules you mean. The
AI crawlers and robots.txt guide lists the bots by purpose
with example rules. If robots.txt allows a bot but it still cannot reach you, check your CDN:
Cloudflare and AI crawlers covers the bot settings that
override robots.txt. The AI crawler checker also fetches the page as
each AI bot.
FAQ
What does a robots.txt validator check?
CrawlWise fetches /robots.txt and reads it the way RFC 9309 describes: a group applies to a crawler only when its user-agent line matches, and the most specific rule wins. It reports the HTTP status, the content type, the number of groups and rules, and whether any rule blocks every path.
Is a missing robots.txt a problem?
A 404 means crawlers may fetch everything, so it does not block anyone. CrawlWise warns about it because you lose the place to declare your sitemap and to set rules for AI crawlers. A 5xx answer is worse: Google treats it as a temporary block on the whole site.
Can I test a URL against my robots.txt rules?
The report tests the site root for each search and training bot, and tests the scanned page's same-origin CSS and JavaScript for Googlebot. To see how a specific page is served to AI bots, the AI crawler checker fetches the page you enter with each bot's user agent.
Does robots.txt stop AI models from using my content?
It asks crawlers that follow it not to fetch pages. It does not remove content that was already collected, and crawlers that ignore the standard are not stopped by it. Blocking training bots while allowing search bots keeps you citable in AI answers.
Related guides
More free checkers
Last updated .