# AI crawlers in robots.txt: what to allow and block Every large AI company now runs several bots, and each one has a different job. One collects pages for model training, another builds the index behind an AI search product, and a third fetches a page when a user asks about it. A single rule for "AI bots" either hides you from AI search or lets in training you did not mean to allow. This guide explains which bot does what, gives you a policy you can copy, and covers the robots.txt rules that quietly break most AI bot policies. ## Three kinds of AI bot - **Training crawlers** collect public pages that may be used to train models. Blocking them keeps your future content out of training data. It does not remove you from AI search. - **Search crawlers** build the index that AI search answers are drawn from. Block them and your pages stop appearing as sources in those answers. - **User-triggered fetchers** load a page because a person asked an assistant about it, for example by pasting your URL into a chat. They do not crawl on their own. The first is a policy decision about your content. The second and third decide whether AI assistants can cite and read you at all. ## The bots that matter | Company | robots.txt token | Job | Follows robots.txt | |---|---|---|---| | OpenAI | `GPTBot` | Training | Yes | | OpenAI | `OAI-SearchBot` | ChatGPT search | Yes | | OpenAI | `ChatGPT-User` | Fetches for a user | "robots.txt rules may not apply" | | Anthropic | `ClaudeBot` | Training | Yes | | Anthropic | `Claude-SearchBot` | Claude search | Yes | | Anthropic | `Claude-User` | Fetches for a user | Yes | | Perplexity | `PerplexityBot` | Perplexity search, not training | Yes | | Perplexity | `Perplexity-User` | Fetches for a user | "generally ignores robots.txt rules" | | Google | `Google-Extended` | Gemini training and grounding | Control token only | | Apple | `Applebot-Extended` | Apple foundation model training | Control token only | | Common Crawl | `CCBot` | Open dataset used for training | Yes | A few details from the vendors' own documentation are easy to miss: - **OpenAI's settings are independent.** You can allow `OAI-SearchBot` and disallow `GPTBot`. A site that blocks `OAI-SearchBot` is left out of ChatGPT search answers, though it can still appear as a plain navigational link. - **Anthropic says all of its bots honour robots.txt**, including `Claude-User`, and supports the non-standard `Crawl-delay` directive. - **`Google-Extended` and `Applebot-Extended` are not crawlers.** Googlebot and Applebot do the fetching; the extended tokens only control how the fetched content may be used. Google says `Google-Extended` does not affect inclusion or ranking in Google Search, and Apple says the same for Applebot-Extended and Apple's search. - **User-triggered fetchers are the exception.** OpenAI and Perplexity both say their fetchers may not apply robots.txt, because a person asked for the page. ## A policy you can copy For most businesses the useful default is: let AI search and user fetchers in, and decide about training separately. This file allows search and user fetches, blocks training, and keeps a private path closed for everyone: ```text # Everyone else User-agent: * Allow: / Disallow: /admin/ # AI search and fetches for users User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: Claude-SearchBot User-agent: Claude-User User-agent: PerplexityBot User-agent: Perplexity-User Allow: / Disallow: /admin/ # Model training User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: CCBot Disallow: / Sitemap: https://example.com/sitemap.xml ``` If you are happy for your content to be used in training, change the last group to `Allow: /` and `Disallow: /admin/`. Naming the bots is still worth it: the policy is explicit, and you can change one group later without touching the others. Blocking training is a legitimate choice and costs you nothing in AI search. Blocking the search crawlers is the expensive mistake: it removes you from the answers people are reading. ## The robots.txt rules that break AI policies robots.txt has been an IETF standard since 2022 (RFC 9309), and a few of its rules surprise almost everyone. ### A named group replaces the `*` group A crawler follows only the group that names it. If no group names it, it follows `*`. Rules do not add up across groups. Here `GPTBot` may read `/admin/`, because its own group says nothing about it: ```text User-agent: * Disallow: /admin/ User-agent: GPTBot Allow: / ``` Repeat every rule that should apply to a named bot inside its own group, as in the policy above. ### Tokens match the product name, not the full user agent `User-agent: GPTBot` matches a request from `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot`. Matching is case-insensitive. Do not put version numbers or full user agent strings in robots.txt; vendors change them. ### The longest matching path wins When several rules match a URL, the one with the longest path wins. If an `Allow` and a `Disallow` are equally long, `Allow` wins. `Allow: /blog/` therefore opens `/blog/` inside a group that says `Disallow: /`. ### A broken robots.txt blocks everyone The status code matters as much as the content: - **4xx**, for example 404: crawlers may treat the whole site as allowed. - **5xx, a timeout or a network error**: crawlers must assume the whole site is disallowed. A robots.txt that errors behind a misconfigured proxy blocks every compliant bot, AI or not. ### Each host needs its own file `https://example.com/robots.txt` does not cover `https://shop.example.com/` or `https://www.example.com/`. Every host and subdomain you want crawled needs its own file. ### Changes take time OpenAI and Perplexity both say a robots.txt change can take about 24 hours to reach their systems. Do not judge a change by the next hour of logs. ## robots.txt is not access control robots.txt is a request that compliant bots honour. It does not stop a client from fetching a page, and it is not the only thing that decides whether a bot gets through. - **Your CDN or firewall can block bots that robots.txt allows.** Cloudflare, for example, lets you allow or block AI search, agent and training crawlers separately at its edge, and can add rules to your robots.txt for you. Its settings are covered in [Is Cloudflare blocking AI crawlers on your site?](/guides/cloudflare-ai-crawlers). Other firewalls and bot managers can block bots too. - **To really block a fetcher, use the firewall.** User-triggered fetchers may ignore robots.txt. OpenAI and Perplexity publish their IP ranges, so you can allow or block them precisely. - **Do not block by IP if you rely on robots.txt.** Anthropic notes that blocking its IP ranges stops its bots from reading your robots.txt, so the opt-out is not recorded reliably. ## Check your site Look at the file the bots see, and at the response code: ```sh curl -sS -o /dev/null -w "%{http_code}\n" https://example.com/robots.txt curl -sS https://example.com/robots.txt ``` Then send a request with a bot's user agent, and compare it with a normal one: ```sh curl -sS -o /dev/null -w "%{http_code}\n" \ -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot" \ https://example.com/ ``` A 403 or a challenge page here usually means a firewall rule. Some bot managers verify bots by IP address, so a spoofed user agent can be treated differently from the real crawler; the firewall's own logs are the final word. A [CrawlWise scan](/) reads your robots.txt the way RFC 9309 describes, evaluates the search bots and the training bots separately, and flags rules that block the CSS and JavaScript a renderer needs. ## Sources - OpenAI: [Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots) - Anthropic: [Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518) - Perplexity: [Perplexity crawlers](https://docs.perplexity.ai/docs/resources/perplexity-crawlers) - Google: [Google's common crawlers: Google-Extended](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) - Apple: [About Applebot](https://support.apple.com/en-us/119829) - IETF: [RFC 9309, Robots Exclusion Protocol](https://datatracker.ietf.org/doc/html/rfc9309) --- Last updated 3 November 2026. HTML version: Every CrawlWise page in Markdown: