AI crawlers in robots.txt: what to allow and block

Every large AI company now runs several bots, and each one has a different job. One collects pages for model training, another builds the index behind an AI search product, and a third fetches a page when a user asks about it. A single rule for “AI bots” either hides you from AI search or lets in training you did not mean to allow.

This guide explains which bot does what, gives you a policy you can copy, and covers the robots.txt rules that quietly break most AI bot policies.

Three kinds of AI bot

• Training crawlers collect public pages that may be used to train models. Blocking them keeps your future content out of training data. It does not remove you from AI search.
• Search crawlers build the index that AI search answers are drawn from. Block them and your pages stop appearing as sources in those answers.
• User-triggered fetchers load a page because a person asked an assistant about it, for example by pasting your URL into a chat. They do not crawl on their own.

The first is a policy decision about your content. The second and third decide whether AI assistants can cite and read you at all.

The bots that matter

A few details from the vendors’ own documentation are easy to miss:

• OpenAI’s settings are independent. You can allow OAI-SearchBot and disallow GPTBot. A site that blocks OAI-SearchBot is left out of ChatGPT search answers, though it can still appear as a plain navigational link.
• Anthropic says all of its bots honour robots.txt, including Claude-User, and supports the non-standard Crawl-delay directive.
• Google-Extended and Applebot-Extended are not crawlers. Googlebot and Applebot do the fetching; the extended tokens only control how the fetched content may be used. Google says Google-Extended does not affect inclusion or ranking in Google Search, and Apple says the same for Applebot-Extended and Apple’s search.
• User-triggered fetchers are the exception. OpenAI and Perplexity both say their fetchers may not apply robots.txt, because a person asked for the page.

A policy you can copy

For most businesses the useful default is: let AI search and user fetchers in, and decide about training separately. This file allows search and user fetches, blocks training, and keeps a private path closed for everyone:

# Everyone else
User-agent: *
Allow: /
Disallow: /admin/

# AI search and fetches for users
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /
Disallow: /admin/

# Model training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

If you are happy for your content to be used in training, change the last group to Allow: / and Disallow: /admin/. Naming the bots is still worth it: the policy is explicit, and you can change one group later without touching the others.

Blocking training is a legitimate choice and costs you nothing in AI search. Blocking the search crawlers is the expensive mistake: it removes you from the answers people are reading.

The robots.txt rules that break AI policies

robots.txt has been an IETF standard since 2022 (RFC 9309), and a few of its rules surprise almost everyone.

A named group replaces the * group

A crawler follows only the group that names it. If no group names it, it follows *. Rules do not add up across groups. Here GPTBot may read /admin/, because its own group says nothing about it:

User-agent: *
Disallow: /admin/

User-agent: GPTBot
Allow: /

Repeat every rule that should apply to a named bot inside its own group, as in the policy above.

Tokens match the product name, not the full user agent

User-agent: GPTBot matches a request from Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot. Matching is case-insensitive. Do not put version numbers or full user agent strings in robots.txt; vendors change them.

The longest matching path wins

When several rules match a URL, the one with the longest path wins. If an Allow and a Disallow are equally long, Allow wins. Allow: /blog/ therefore opens /blog/ inside a group that says Disallow: /.

A broken robots.txt blocks everyone

The status code matters as much as the content:

• 4xx, for example 404: crawlers may treat the whole site as allowed.
• 5xx, a timeout or a network error: crawlers must assume the whole site is disallowed.

A robots.txt that errors behind a misconfigured proxy blocks every compliant bot, AI or not.

Each host needs its own file

https://example.com/robots.txt does not cover https://shop.example.com/ or https://www.example.com/. Every host and subdomain you want crawled needs its own file.

Changes take time

OpenAI and Perplexity both say a robots.txt change can take about 24 hours to reach their systems. Do not judge a change by the next hour of logs.

robots.txt is not access control

robots.txt is a request that compliant bots honour. It does not stop a client from fetching a page, and it is not the only thing that decides whether a bot gets through.

• Your CDN or firewall can block bots that robots.txt allows. Cloudflare, for example, lets you allow or block AI search, agent and training crawlers separately at its edge, and can add rules to your robots.txt for you. Its settings are covered in Is Cloudflare blocking AI crawlers on your site?. Other firewalls and bot managers can block bots too.
• To really block a fetcher, use the firewall. User-triggered fetchers may ignore robots.txt. OpenAI and Perplexity publish their IP ranges, so you can allow or block them precisely.
• Do not block by IP if you rely on robots.txt. Anthropic notes that blocking its IP ranges stops its bots from reading your robots.txt, so the opt-out is not recorded reliably.

Check your site

Look at the file the bots see, and at the response code:

curl -sS -o /dev/null -w "%{http_code}\n" https://example.com/robots.txt
curl -sS https://example.com/robots.txt

Then send a request with a bot’s user agent, and compare it with a normal one:

curl -sS -o /dev/null -w "%{http_code}\n" \
  -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot" \
  https://example.com/

A 403 or a challenge page here usually means a firewall rule. Some bot managers verify bots by IP address, so a spoofed user agent can be treated differently from the real crawler; the firewall’s own logs are the final word.

A CrawlWise scan reads your robots.txt the way RFC 9309 describes, evaluates the search bots and the training bots separately, and flags rules that block the CSS and JavaScript a renderer needs.

Sources

• OpenAI: Overview of OpenAI crawlers
• Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?
• Perplexity: Perplexity crawlers
• Google: Google’s common crawlers: Google-Extended
• Apple: About Applebot
• IETF: RFC 9309, Robots Exclusion Protocol