# Cloudflare rules to block or allow AI crawlers and agents

> Cloudflare rules to block all AI crawlers and agents, or let search bots in: the dashboard toggle, custom rule expressions, robots.txt limits and how to test.

Canonical: https://red.run.cloud/learn/cloudflare-rules-block-or-allow-ai-crawlers-and-agents · By Timothy Lindblom · Published 2026-11-02 · Updated 2026-11-02

To block all AI crawlers and agents in Cloudflare, open Security Settings, choose Configure AI bot policies and set Search, Training and Agent to block. Then add a custom rule for the user agents that policy misses. Two traps remain. robots.txt is voluntary, and a rule that is too wide can block Googlebot.

| Fact | Detail | Source |
|---|---|---|
| Dashboard path | Security Settings, then Configure AI bot policies; the older toggle is Block AI bots | [Cloudflare](https://developers.cloudflare.com/bots/additional-configurations/block-ai-bots/) |
| Three policies | Search, Training, Agent. Each can block across the zone, block on pages with ads, or allow | Cloudflare |
| Default change | From 15 September 2026, new domains block Training and Agent bots by default on pages with ads | Cloudflare |
| Per-crawler control | AI Crawl Control, available on all plans | [Cloudflare](https://developers.cloudflare.com/ai-crawl-control/) |
| Free plan custom rules | 5 rules, every action except Log | [Cloudflare](https://developers.cloudflare.com/waf/custom-rules/) |
| robots.txt | Voluntary; the file says what you prefer and enforces nothing | [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309) |
| Checked | 24 October 2026 | this page |

## What is the difference between an AI crawler and an AI agent?

A crawler fetches pages on its own schedule. An agent fetches a page because a person asked a question or gave a task. Cloudflare's [verified bots documentation](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/) uses the same split: Search builds indexes, Training feeds model development, and Agent covers user-directed visits.

The difference decides which tool works. A crawler that follows robots.txt can be asked to leave. A user-triggered fetch often ignores robots.txt, so you need a firewall rule.

| Bot | Behaviour | Follows robots.txt? | Source |
|---|---|---|---|
| GPTBot | Training | Yes | [OpenAI](https://developers.openai.com/api/docs/bots) |
| OAI-SearchBot | Search results in ChatGPT | Yes | OpenAI |
| ChatGPT-User | User action in ChatGPT | OpenAI says rules may not apply | OpenAI |
| ClaudeBot | Training | Yes, per Anthropic's opt-out guidance | [Anthropic](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) |
| Claude-SearchBot | Search quality | Listed as a separate agent | Anthropic |
| Claude-User | User questions | Listed as a separate agent | Anthropic |
| PerplexityBot | Search results, not model training | Yes | [Perplexity](https://docs.perplexity.ai/guides/bots) |
| Perplexity-User | User questions | Generally ignores it | Perplexity |
| Google-Extended | Gemini training and grounding | A robots.txt token only | [Google](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) |

Table status at our check on 24 October 2026. Vendors add and rename agents, so open the linked page before you copy a name.

<figure class="ln-figure">
<svg viewBox="0 0 720 300" role="img" xmlns="http://www.w3.org/2000/svg" font-family="Geist, sans-serif">
<title>Decision flow: who is asking decides which control works</title>
<rect x="0" y="0" width="720" height="300" fill="#FFFFFF"/>
<rect x="250" y="16" width="220" height="44" rx="10" fill="#1A0F0F"/>
<text x="360" y="43" text-anchor="middle" font-size="15" fill="#FFFFFF">A request arrives</text>
<rect x="20" y="110" width="200" height="64" rx="10" fill="#FFF1EF" stroke="#EAD3CF"/>
<text x="120" y="136" text-anchor="middle" font-size="14" fill="#1A0F0F">Training crawler</text>
<text x="120" y="156" text-anchor="middle" font-size="12" fill="#655553">GPTBot, ClaudeBot</text>
<rect x="260" y="110" width="200" height="64" rx="10" fill="#FFF1EF" stroke="#EAD3CF"/>
<text x="360" y="136" text-anchor="middle" font-size="14" fill="#1A0F0F">Search crawler</text>
<text x="360" y="156" text-anchor="middle" font-size="12" fill="#655553">OAI-SearchBot, PerplexityBot</text>
<rect x="500" y="110" width="200" height="64" rx="10" fill="#FFF1EF" stroke="#EAD3CF"/>
<text x="600" y="136" text-anchor="middle" font-size="14" fill="#1A0F0F">Agent for a person</text>
<text x="600" y="156" text-anchor="middle" font-size="12" fill="#655553">ChatGPT-User, Perplexity-User</text>
<path d="M360 60 L120 110 M360 60 L360 110 M360 60 L600 110" stroke="#655553" stroke-width="1.5" fill="none"/>
<rect x="20" y="210" width="200" height="64" rx="10" fill="#FFFFFF" stroke="#C70500"/>
<text x="120" y="236" text-anchor="middle" font-size="13" fill="#B00500">robots.txt plus policy</text>
<text x="120" y="256" text-anchor="middle" font-size="12" fill="#655553">block if you want out</text>
<rect x="260" y="210" width="200" height="64" rx="10" fill="#FFFFFF" stroke="#C70500"/>
<text x="360" y="236" text-anchor="middle" font-size="13" fill="#B00500">Allow to be listed</text>
<text x="360" y="256" text-anchor="middle" font-size="12" fill="#655553">block to leave AI search</text>
<rect x="500" y="210" width="200" height="64" rx="10" fill="#FFFFFF" stroke="#C70500"/>
<text x="600" y="236" text-anchor="middle" font-size="13" fill="#B00500">Firewall rule only</text>
<text x="600" y="256" text-anchor="middle" font-size="12" fill="#655553">robots.txt may be ignored</text>
<path d="M120 174 L120 210 M360 174 L360 210 M600 174 L600 210" stroke="#655553" stroke-width="1.5" fill="none"/>
</svg>
<figcaption>Who is asking decides which control works. Sources: OpenAI, Anthropic and Perplexity crawler pages, checked 24 October 2026.</figcaption>
</figure>

## How do I block AI bots with Cloudflare's own settings?

Start with the dashboard, because it is maintained by Cloudflare as bots change. In [Cloudflare's instructions](https://developers.cloudflare.com/bots/additional-configurations/block-ai-bots/), the path is Security Settings, then Configure AI bot policies. Each of Search, Training and Agent takes one of three values: block across the zone, block only on pages that show ads, or allow.

Cloudflare says each policy covers verified bots with that behaviour plus some unverified bots that fit the classification. It also says the older Block AI bots option leaves out mixed-purpose bots used for both Training and Search, and that the newer policy blocks them.

- [ ] Decide per behaviour: Training, Search, Agent. Write the decision down with the date.
- [ ] Set the three policies in Configure AI bot policies.
- [ ] If you used the legacy Block AI bots toggle, move to the policies so mixed-purpose bots are covered.
- [ ] Open [AI Crawl Control](https://developers.cloudflare.com/ai-crawl-control/) to allow or block individual crawlers and see who has been asking.
- [ ] Re-check after 15 September 2026 defaults if the domain is new and shows ads.

Cloudflare's overview says AI Crawl Control is available on all plans, including free. Its page was last updated on 14 August 2026.

## Which custom rules block AI crawlers when the toggle is not enough?

Use a WAF custom rule when you want a named list, a path-specific rule, or a rule that lives in your own infrastructure code. [Custom rules](https://developers.cloudflare.com/waf/custom-rules/) on a Free plan allow 5 rules and every action except Log, so spend them carefully.

The expression below matches by user agent and works on any plan. The names come from the vendor pages in the table above. Paste it into the expression editor and choose Block, or Managed Challenge if you want a softer test first.

```
(http.user_agent contains "GPTBot") or
(http.user_agent contains "ClaudeBot") or
(http.user_agent contains "PerplexityBot")
```

A category rule is shorter and follows Cloudflare's own classification. Cloudflare documents the field `cf.verified_bot_category` with the legacy values `AI Crawler`, `AI Assistant` and `AI Search`, and notes that from July 2026 AI Search is consolidated with traditional search under the Search behaviour. Do not assume the old values still match what you want. Build the expression in the editor, which validates field names for your plan, and read [the category list](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/categories/) first.

A category that includes search crawlers can catch Googlebot or Bingbot. Cloudflare lists Search Engine Crawler as its own category with Googlebot and Bingbot as examples, so keep that category out of any block rule. If Googlebot already gets a 403, [how to find which layer blocked it](/learn/cloudflare-bot-fight-mode-blocked-googlebot) is on its own page.

## Which settings should I use for my goal?

| Your goal | Training | Search | Agent | Note |
|---|---|---|---|---|
| Keep content out of model training, stay listed in AI search | Block | Allow | Allow | The most common choice for publishers |
| Leave AI search too, keep Google | Block | Block | Allow or block | Check that Googlebot is untouched |
| Store with prices and stock you do not want read by agents | Block | Allow | Block | Agents are the hardest to identify |
| Docs or marketing site that wants AI citations | Allow | Allow | Allow | Watch load in your analytics |
| Pages behind login or with customer data | Block | Block | Block | Protect these with authentication; bot rules are a second layer |

The other [AI crawler guides](/learn/topics/ai-crawlers) go deeper on each bot. Allowing Search is what keeps search bots such as OAI-SearchBot able to read your pages. If you turn on a blanket block and then drop out of AI answers, check your own Search policy first.

## Can robots.txt do the job instead?

For well-behaved crawlers it is the polite option, and Anthropic's guidance for site owners is to use robots.txt and not IP blocking, because blocking IPs can stop the crawler reading your opt-out. OpenAI says changes can take about 24 hours to take effect for its systems.

It enforces nothing. Cloudflare's [managed robots.txt page](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/) says compliance is voluntary and that for enforcement you need AI Crawl Control. [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309) says the same in its own words: the rules are not a form of access authorization.

Cloudflare's managed robots.txt can write the directives for you from your policies. Turn on Bot Preference Sync after setting Training to Disallow. It adds Content Signals and Disallow lines for known AI bots such as ClaudeBot and GPTBot.

Google-Extended is a special case, and [llms.txt restricts nothing either](/learn/llms-txt-restricts-nothing). Google says it has no separate user agent string and does not affect inclusion in Search or ranking. It is a robots.txt token only, so a WAF user agent rule cannot enforce it.

## How are agents different, and what is Web Bot Auth?

An agent acts for one person at a time, so you cannot reason about it as a crawler. Google's own agent has [its own verify-or-block page](/learn/google-agent-block-allow-or-verify), and shop owners can read [whether a store should let shopping agents in](/learn/should-your-store-let-ai-shopping-agents-in). Cloudflare's [signed agents page](https://developers.cloudflare.com/bots/concepts/bot/signed-agents/) says signed agents became Verified on 1 July 2026 and separates Direct bots (one operator) from Intermediary ones (a service many users can drive).

[Web Bot Auth](https://developers.cloudflare.com/bots/reference/bot-verification/web-bot-auth/) is the technical basis: a bot signs its requests with a key and publishes the public key, and Cloudflare verifies the signature. Not every agent signs, so unsigned agents fall back to user agent strings and IP lists. Perplexity, for example, publishes IP lists for both its agents and recommends combining user agent matching with IP verification.

A rule that says "block agents" therefore covers the agents Cloudflare can identify. It does not cover a browser automation tool that looks like a person.

## How do I test the rules and avoid blocking Google?

Test before you trust a rule, and test the opposite direction too.

- [ ] Send a request from your own machine with a crawler user agent and confirm a block or challenge response. Use a site you own.
- [ ] Open Security Events and filter by the rule name. Confirm the matches are the bots you expected.
- [ ] Use the URL Inspection live test in Google Search Console and fetch your sitemap, to confirm Googlebot still gets a 200.
- [ ] If you are on the Free plan and see unexplained blocks, check Bot Fight Mode. [Cloudflare states](https://developers.cloudflare.com/bots/get-started/free/) it cannot be bypassed or skipped with WAF custom rules, and that it can challenge API or mobile app traffic.
- [ ] Re-check after 24 hours for OpenAI and again after a week for traffic in your analytics.

A user agent test proves your rule matches that string. It does not prove a real crawler is blocked, because real crawlers arrive from their own addresses and are identified by verification.

## What will a firewall rule not do?

It does not remove content that has already been collected, and it does not make private pages private. If a page should not be read by anyone outside your team, it needs authentication.

A bot rule also does not hide your origin. If your server answers requests sent straight to its IP address, a crawler that finds it never meets Cloudflare. Cloudflare does not shield [a directly reachable origin server](/learn/cloudflare-doesnt-protect-your-origin-ip) by default, and [a WAF is a stopgap](/learn/does-a-waf-stop-php2shell), not a patch. Crawlers that ask for sensitive files are a different problem; see [bots requesting .env and .git](/learn/bots-requesting-env-and-git-30-second-check).

An outside scan can read what your site serves. It cannot see your Cloudflare rules or tell you whether a bot would be blocked; the guide to [what outside scans miss](/learn/what-an-external-website-scan-cant-prove) explains the limit.

## Questions people ask

### How do I block all AI crawlers in Cloudflare?

Open Security Settings, then Configure AI bot policies, and set Search, Training and Agent to block. That covers verified AI bots and some unverified ones. Add a custom rule on user agents for anything the policy misses, and test with a request that sends a crawler's user agent.

### If I block AI bots, will Google still index my site?

Yes, as long as Googlebot is not blocked. Google says its Google-Extended token does not affect inclusion in Google Search or ranking. Be careful with the Search policy and with category rules, because a rule that is too wide can catch search crawlers as well.

### Does robots.txt stop ChatGPT and Perplexity from reading my pages?

It stops the crawlers that choose to follow it. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them. Those fetches are triggered by a person. Only a firewall rule can refuse them.

### Why is my site invisible in AI answers after I turned on Cloudflare's AI block?

The Search policy or a category rule may be blocking the bots that build AI search results, such as OAI-SearchBot. Blocking training crawlers and allowing search crawlers are separate settings. Check the policy for each behaviour and the Security Events log for blocked AI bots.

### Can I block AI agents by user agent alone?

You can, but a user agent string is easy to fake in both directions. Cloudflare verifies bots through signatures, published IP lists or reverse DNS. Use user agent rules as a second layer behind the verified-bot policies, and do not treat them as proof of who sent a request.


## Sources

1. [Cloudflare: Block AI bots](https://developers.cloudflare.com/bots/additional-configurations/block-ai-bots/): Dashboard path, the Search, Training and Agent policies, the legacy toggle's gap, the 15 September 2026 default change. Checked 2 November 2026.
2. [Cloudflare: AI Crawl Control overview](https://developers.cloudflare.com/ai-crawl-control/): Per-crawler allow or block rules, available on all plans. Page updated 14 August 2026.
3. [Cloudflare: Managed robots.txt](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/): Says robots.txt compliance is voluntary.
4. [Cloudflare: Verified bots](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/): How a bot becomes verified; Search, Agent and Training behaviours; Direct and Intermediary.
5. [Cloudflare: Verified bot categories](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/categories/): The legacy cf.verified_bot_category values and the July 2026 consolidation of AI Search.
6. [Cloudflare: Signed agents](https://developers.cloudflare.com/bots/concepts/bot/signed-agents/): Signed agents became Verified on 1 July 2026.
7. [Cloudflare: Web Bot Auth](https://developers.cloudflare.com/bots/reference/bot-verification/web-bot-auth/)
8. [Cloudflare: WAF custom rules](https://developers.cloudflare.com/waf/custom-rules/): Actions and per-plan rule limits.
9. [Cloudflare: Bot Fight Mode](https://developers.cloudflare.com/bots/get-started/free/): States that Bot Fight Mode cannot be skipped with custom rules.
10. [OpenAI: Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots): OAI-SearchBot, GPTBot, ChatGPT-User and OAI-AdsBot, and which follow robots.txt.
11. [Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)
12. [Perplexity: Perplexity crawlers](https://docs.perplexity.ai/guides/bots)
13. [Google: Common crawlers (Google-Extended)](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers)
14. [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309): Section 1.3: the rules are not a form of access authorization.
