# robots.txt checker

Check that /robots.txt answers as plain text, parses into rules, does not block the whole site, and lets search engines and AI search bots fetch the pages and files they need.

Run it at https://crawlwise.dev/tools/robots-txt-checker: paste a public URL, no account needed. The scan runs the full CrawlWise audit and returns a private report.

## What this checker tests

- robots.txt is reachable and valid (`discovery.robots`)
- AI search and citation bots are allowed (`discovery.ai-bots`)
- robots.txt allows CSS and JavaScript needed for rendering (`discovery.resource-access`)

## What the report shows

CrawlWise fetches `/robots.txt` from the scanned host and reports:

- the HTTP status and content type. A `401`, `403` or `429` is reported as **blocked**, not as a
  failure, because the scanner could not read the file. A server error or an HTML page served
  at `/robots.txt` fails;
- how many user-agent groups and rules the file has, and whether a rule disallows every path for
  all crawlers;
- which search and citation bots (`OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot`,
  `Googlebot`, `Bingbot`) and which training bots (`GPTBot`, `ClaudeBot`, `Google-Extended`,
  `CCBot` and others) may fetch the site root;
- whether Googlebot may fetch the page's same-origin stylesheets and scripts, which Google needs
  to render the page.

## Common mistakes

- **An HTML page at /robots.txt.** A single-page app that answers every path with its shell
  gives crawlers a file without rules.
- **`Disallow: /` left over from staging.** One line blocks the whole site for every crawler
  that reads the `*` group.
- **Blocking `/assets/` or `/_next/`.** Crawlers can then fetch the HTML but not the CSS and
  JavaScript that render it.
- **Blocking every AI bot at once.** A rule for `GPTBot` does not cover `OAI-SearchBot`. Decide
  on search and training bots separately.

## Fix it

Serve `/robots.txt` as `text/plain` with a `Sitemap:` line and only the rules you mean. The
[AI crawlers and robots.txt guide](/guides/ai-crawlers-robots-txt) lists the bots by purpose
with example rules. If robots.txt allows a bot but it still cannot reach you, check your CDN:
[Cloudflare and AI crawlers](/guides/cloudflare-ai-crawlers) covers the bot settings that
override robots.txt. The [AI crawler checker](/tools/ai-crawler-checker) also fetches the page as
each AI bot.

## FAQ

### What does a robots.txt validator check?

CrawlWise fetches /robots.txt and reads it the way RFC 9309 describes: a group applies to a crawler only when its user-agent line matches, and the most specific rule wins. It reports the HTTP status, the content type, the number of groups and rules, and whether any rule blocks every path.

### Is a missing robots.txt a problem?

A 404 means crawlers may fetch everything, so it does not block anyone. CrawlWise warns about it because you lose the place to declare your sitemap and to set rules for AI crawlers. A 5xx answer is worse: Google treats it as a temporary block on the whole site.

### Can I test a URL against my robots.txt rules?

The report tests the site root for each search and training bot, and tests the scanned page's same-origin CSS and JavaScript for Googlebot. To see how a specific page is served to AI bots, the AI crawler checker fetches the page you enter with each bot's user agent.

### Does robots.txt stop AI models from using my content?

It asks crawlers that follow it not to fetch pages. It does not remove content that was already collected, and crawlers that ignore the standard are not stopped by it. Blocking training bots while allowing search bots keeps you citable in AI answers.

## Related guides

- [AI crawlers in robots.txt: what to allow and block](https://crawlwise.dev/guides/ai-crawlers-robots-txt.md)
- [Is Cloudflare blocking AI crawlers on your site?](https://crawlwise.dev/guides/cloudflare-ai-crawlers.md)

---

Last updated 19 November 2026.

HTML version: <https://crawlwise.dev/tools/robots-txt-checker>

Every CrawlWise page in Markdown: <https://crawlwise.dev/llms.txt>
