# Is Cloudflare blocking AI crawlers on your site? Cloudflare answers requests before they reach your server. If it blocks a crawler, your robots.txt never gets a say: the bot receives a 403 or a 402 at the edge, and the request never shows up in your own logs. A site can allow every AI bot in robots.txt and still be invisible to AI search because of one setting in the Cloudflare dashboard. This guide covers Cloudflare's AI crawler settings as documented in November 2026. They have changed several times since 2025, so check the linked documentation before relying on a detail. ## How Cloudflare groups AI crawlers Cloudflare sorts AI traffic by what the bot does: - **Search**: collects or indexes your content so it can answer questions about it later. - **Agent**: acts in real time on a person's behalf, such as chat fetchers and browser-use agents. - **Training**: crawls content to train or fine-tune a model. Some crawlers do two jobs. Cloudflare calls a crawler that does both search and training a **mixed-use crawler**, and names Googlebot, Bingbot and Applebot as examples. That detail matters more than any other on this page. Each policy applies to verified bots with that behaviour and to unverified bots that claim to be them. ## Where the settings are ### AI Crawler and Agent access policies In the Cloudflare dashboard, go to **Security** > **Settings**, filter by **Bot traffic**, and open **AI Crawler and Agent access policies**. Search, Agent and Training each get one action: | Action | Effect | |---|---| | Allow (do not block) | Requests pass through. | | Block | Requests are blocked across the zone. | | Block on pages with ads | Requests are blocked only on pages where Cloudflare detects ads. | | Disallow | Training only: tells crawlers in robots.txt not to use your content. | Cloudflare's documentation says new domains start with all three set to **Allow**. Since September 2026, the onboarding flow also offers presets depending on whether the site shows ads, so an existing zone may have been set up differently. ### AI Crawl Control **AI Crawl Control**, available on all plans, lists the AI crawlers that visit your site, how many requests each one made, and how often each ignored your robots.txt. You can allow or block crawlers one by one, and on paid plans choose whether blocked crawlers receive **403 Forbidden** or **402 Payment Required**. Blocking here creates a WAF custom rule on the zone. ### Bot Preference Sync With **Enable Bot Preference Sync** turned on and at least one policy set to Block, Block on pages with ads or Disallow, Cloudflare adds managed rules to the top of your robots.txt. That block names training crawlers such as `GPTBot`, `ClaudeBot`, `CCBot` and `Google-Extended`, and can include a `Content-signal` line such as: ```text Content-signal: search=yes, ai-train=no ``` Your own rules stay below the managed block. When every policy is set to Allow, robots.txt is served unchanged. ## The setting that can remove you from Google To stop AI training, the obvious choice is **Training: Block**. It is the wrong one for most sites. Since September 2026, Block and Block on pages with ads apply to mixed-use crawlers as well. Cloudflare's own words: Block "will stop Applebot, Bingbot, and Googlebot from reaching your site — search included." On ad-supported pages, Block on pages with ads does the same to those pages. To stop training and keep search, choose **Disallow** with Bot Preference Sync enabled. Cloudflare then publishes the no-training preference in robots.txt and blocks training-only crawlers, such as those run by Amazon, Anthropic, Meta and OpenAI, at the edge. Mixed-use crawlers stay allowed for search. Cloudflare notes that this preference does not yet reach Bing through robots.txt. ## A setup that keeps you in AI search For a business that wants to be found and cited: 1. **Search: Allow.** AI search crawlers such as `OAI-SearchBot`, `Claude-SearchBot` and `PerplexityBot` build the indexes AI answers draw on. 2. **Agent: Allow.** These are the fetchers that read your page when a user asks about it. 3. **Training: Allow or Disallow.** Choose Disallow if you do not want your content used for training. Avoid Block unless you also mean to leave Google, Bing and Apple search. Then review **AI Crawl Control** for crawlers blocked one by one, and the WAF custom rules for anything that matches bot user agents. ## Check what crawlers get **Read the robots.txt Cloudflare serves**, not the file in your repository. Managed rules are added at the edge: ```sh curl -sS https://example.com/robots.txt | head -40 ``` **Send a request with a crawler's user agent:** ```sh curl -sS -o /dev/null -w "%{http_code}\n" \ -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot" \ https://example.com/ ``` A 403 or 402 means a rule matched. A 200 does not prove the real crawler gets through: your request comes from your own IP, so Cloudflare treats it as an unverified bot, while the real crawler is verified by its IP range or signature. Use this test to find blocks, not to rule them out. **Look at Cloudflare's own data.** AI Crawl Control shows requests per crawler and the action taken, and Cloudflare's security events log shows individual blocked requests and the rule that matched each one. A [CrawlWise scan](/) reads the robots.txt your site serves, including any managed rules Cloudflare adds, and reports which AI search and training bots it allows. It also fetches the page with the user agents of six AI crawlers and reports any that get an error, a challenge or less content than a regular request. Like the `curl` test above, those requests come from CrawlWise, not from the crawlers, so check the dashboard as well. ## Pay per crawl Cloudflare also runs pay per crawl, in closed beta: a crawler that is set to be charged receives `402 Payment Required` with a price and can retry with payment. If you have not joined it, the practical question is only whether you allow or block, as above. ## Sources - Cloudflare: [AI Crawler and Agent access policies](https://developers.cloudflare.com/bots/additional-configurations/block-ai-bots/) - Cloudflare: [Have it both ways: stay discoverable in search while disallowing AI training](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/) - Cloudflare: [AI Crawl Control](https://developers.cloudflare.com/ai-crawl-control/) - Cloudflare: [Managed robots.txt and Bot Preference Sync](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/) - Cloudflare: [Bot categories](https://developers.cloudflare.com/bots/concepts/bot/) - Cloudflare: [What is pay per crawl?](https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/) --- Last updated 4 November 2026. HTML version: Every CrawlWise page in Markdown: