CrawlWise

Guides

Crawler access

Is Cloudflare blocking AI crawlers on your site?

Cloudflare can block AI search, agent and training crawlers before they reach your server. Where the settings are, what each one does, and how to check.

Cloudflare answers requests before they reach your server. If it blocks a crawler, your robots.txt never gets a say: the bot receives a 403 or a 402 at the edge, and the request never shows up in your own logs. A site can allow every AI bot in robots.txt and still be invisible to AI search because of one setting in the Cloudflare dashboard.

This guide covers Cloudflare’s AI crawler settings as documented in November 2026. They have changed several times since 2025, so check the linked documentation before relying on a detail.

How Cloudflare groups AI crawlers

Cloudflare sorts AI traffic by what the bot does:

  • Search: collects or indexes your content so it can answer questions about it later.
  • Agent: acts in real time on a person’s behalf, such as chat fetchers and browser-use agents.
  • Training: crawls content to train or fine-tune a model.

Some crawlers do two jobs. Cloudflare calls a crawler that does both search and training a mixed-use crawler, and names Googlebot, Bingbot and Applebot as examples. That detail matters more than any other on this page.

Each policy applies to verified bots with that behaviour and to unverified bots that claim to be them.

Where the settings are

AI Crawler and Agent access policies

In the Cloudflare dashboard, go to Security > Settings, filter by Bot traffic, and open AI Crawler and Agent access policies. Search, Agent and Training each get one action:

Action Effect
Allow (do not block) Requests pass through.
Block Requests are blocked across the zone.
Block on pages with ads Requests are blocked only on pages where Cloudflare detects ads.
Disallow Training only: tells crawlers in robots.txt not to use your content.

Cloudflare’s documentation says new domains start with all three set to Allow. Since September 2026, the onboarding flow also offers presets depending on whether the site shows ads, so an existing zone may have been set up differently.

AI Crawl Control

AI Crawl Control, available on all plans, lists the AI crawlers that visit your site, how many requests each one made, and how often each ignored your robots.txt. You can allow or block crawlers one by one, and on paid plans choose whether blocked crawlers receive 403 Forbidden or 402 Payment Required. Blocking here creates a WAF custom rule on the zone.

Bot Preference Sync

With Enable Bot Preference Sync turned on and at least one policy set to Block, Block on pages with ads or Disallow, Cloudflare adds managed rules to the top of your robots.txt. That block names training crawlers such as GPTBot, ClaudeBot, CCBot and Google-Extended, and can include a Content-signal line such as:

Content-signal: search=yes, ai-train=no

Your own rules stay below the managed block. When every policy is set to Allow, robots.txt is served unchanged.

The setting that can remove you from Google

To stop AI training, the obvious choice is Training: Block. It is the wrong one for most sites.

Since September 2026, Block and Block on pages with ads apply to mixed-use crawlers as well. Cloudflare’s own words: Block “will stop Applebot, Bingbot, and Googlebot from reaching your site — search included.” On ad-supported pages, Block on pages with ads does the same to those pages.

To stop training and keep search, choose Disallow with Bot Preference Sync enabled. Cloudflare then publishes the no-training preference in robots.txt and blocks training-only crawlers, such as those run by Amazon, Anthropic, Meta and OpenAI, at the edge. Mixed-use crawlers stay allowed for search. Cloudflare notes that this preference does not yet reach Bing through robots.txt.

For a business that wants to be found and cited:

  1. Search: Allow. AI search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot build the indexes AI answers draw on.
  2. Agent: Allow. These are the fetchers that read your page when a user asks about it.
  3. Training: Allow or Disallow. Choose Disallow if you do not want your content used for training. Avoid Block unless you also mean to leave Google, Bing and Apple search.

Then review AI Crawl Control for crawlers blocked one by one, and the WAF custom rules for anything that matches bot user agents.

Check what crawlers get

Read the robots.txt Cloudflare serves, not the file in your repository. Managed rules are added at the edge:

curl -sS https://example.com/robots.txt | head -40

Send a request with a crawler’s user agent:

curl -sS -o /dev/null -w "%{http_code}\n" \
  -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot" \
  https://example.com/

A 403 or 402 means a rule matched. A 200 does not prove the real crawler gets through: your request comes from your own IP, so Cloudflare treats it as an unverified bot, while the real crawler is verified by its IP range or signature. Use this test to find blocks, not to rule them out.

Look at Cloudflare’s own data. AI Crawl Control shows requests per crawler and the action taken, and Cloudflare’s security events log shows individual blocked requests and the rule that matched each one.

A CrawlWise scan reads the robots.txt your site serves, including any managed rules Cloudflare adds, and reports which AI search and training bots it allows. It also fetches the page with the user agents of six AI crawlers and reports any that get an error, a challenge or less content than a regular request. Like the curl test above, those requests come from CrawlWise, not from the crawlers, so check the dashboard as well.

Pay per crawl

Cloudflare also runs pay per crawl, in closed beta: a crawler that is set to be charged receives 402 Payment Required with a price and can retry with payment. If you have not joined it, the practical question is only whether you allow or block, as above.

Sources

How CrawlWise checks this

Scan your page

Last updated .