1. Red
  2. Learn
  3. AI crawlers and bots

Cloudflare rules to block or allow AI crawlers and agents

Cloudflare rules to block all AI crawlers and agents, or let search bots in: the dashboard toggle, custom rule expressions, robots.txt limits and how to test.

By Timothy Lindblom, FounderPublished 2 November 20267 min readHow we check facts
On this page
  1. What is the difference between an AI crawler and an AI agent?
  2. How do I block AI bots with Cloudflare's own settings?
  3. Which custom rules block AI crawlers when the toggle is not enough?
  4. Which settings should I use for my goal?
  5. Can robots.txt do the job instead?
  6. How are agents different, and what is Web Bot Auth?
  7. How do I test the rules and avoid blocking Google?
  8. What will a firewall rule not do?

To block all AI crawlers and agents in Cloudflare, open Security Settings, choose Configure AI bot policies and set Search, Training and Agent to block. Then add a custom rule for the user agents that policy misses. Two traps remain. robots.txt is voluntary, and a rule that is too wide can block Googlebot.

FactDetailSource
Dashboard pathSecurity Settings, then Configure AI bot policies; the older toggle is Block AI botsCloudflare
Three policiesSearch, Training, Agent. Each can block across the zone, block on pages with ads, or allowCloudflare
Default changeFrom 15 September 2026, new domains block Training and Agent bots by default on pages with adsCloudflare
Per-crawler controlAI Crawl Control, available on all plansCloudflare
Free plan custom rules5 rules, every action except LogCloudflare
robots.txtVoluntary; the file says what you prefer and enforces nothingRFC 9309
Checked24 October 2026this page

What is the difference between an AI crawler and an AI agent?

A crawler fetches pages on its own schedule. An agent fetches a page because a person asked a question or gave a task. Cloudflare's verified bots documentation uses the same split: Search builds indexes, Training feeds model development, and Agent covers user-directed visits.

The difference decides which tool works. A crawler that follows robots.txt can be asked to leave. A user-triggered fetch often ignores robots.txt, so you need a firewall rule.

BotBehaviourFollows robots.txt?Source
GPTBotTrainingYesOpenAI
OAI-SearchBotSearch results in ChatGPTYesOpenAI
ChatGPT-UserUser action in ChatGPTOpenAI says rules may not applyOpenAI
ClaudeBotTrainingYes, per Anthropic's opt-out guidanceAnthropic
Claude-SearchBotSearch qualityListed as a separate agentAnthropic
Claude-UserUser questionsListed as a separate agentAnthropic
PerplexityBotSearch results, not model trainingYesPerplexity
Perplexity-UserUser questionsGenerally ignores itPerplexity
Google-ExtendedGemini training and groundingA robots.txt token onlyGoogle

Table status at our check on 24 October 2026. Vendors add and rename agents, so open the linked page before you copy a name.

Decision flow: who is asking decides which control works A request arrives Training crawler GPTBot, ClaudeBot Search crawler OAI-SearchBot, PerplexityBot Agent for a person ChatGPT-User, Perplexity-User robots.txt plus policy block if you want out Allow to be listed block to leave AI search Firewall rule only robots.txt may be ignored
Who is asking decides which control works. Sources: OpenAI, Anthropic and Perplexity crawler pages, checked 24 October 2026.

How do I block AI bots with Cloudflare's own settings?

Start with the dashboard, because it is maintained by Cloudflare as bots change. In Cloudflare's instructions, the path is Security Settings, then Configure AI bot policies. Each of Search, Training and Agent takes one of three values: block across the zone, block only on pages that show ads, or allow.

Cloudflare says each policy covers verified bots with that behaviour plus some unverified bots that fit the classification. It also says the older Block AI bots option leaves out mixed-purpose bots used for both Training and Search, and that the newer policy blocks them.

  • Decide per behaviour: Training, Search, Agent. Write the decision down with the date.
  • Set the three policies in Configure AI bot policies.
  • If you used the legacy Block AI bots toggle, move to the policies so mixed-purpose bots are covered.
  • Open AI Crawl Control to allow or block individual crawlers and see who has been asking.
  • Re-check after 15 September 2026 defaults if the domain is new and shows ads.

Cloudflare's overview says AI Crawl Control is available on all plans, including free. Its page was last updated on 14 August 2026.

Which custom rules block AI crawlers when the toggle is not enough?

Use a WAF custom rule when you want a named list, a path-specific rule, or a rule that lives in your own infrastructure code. Custom rules on a Free plan allow 5 rules and every action except Log, so spend them carefully.

The expression below matches by user agent and works on any plan. The names come from the vendor pages in the table above. Paste it into the expression editor and choose Block, or Managed Challenge if you want a softer test first.

(http.user_agent contains "GPTBot") or
(http.user_agent contains "ClaudeBot") or
(http.user_agent contains "PerplexityBot")

A category rule is shorter and follows Cloudflare's own classification. Cloudflare documents the field cf.verified_bot_category with the legacy values AI Crawler, AI Assistant and AI Search, and notes that from July 2026 AI Search is consolidated with traditional search under the Search behaviour. Do not assume the old values still match what you want. Build the expression in the editor, which validates field names for your plan, and read the category list first.

A category that includes search crawlers can catch Googlebot or Bingbot. Cloudflare lists Search Engine Crawler as its own category with Googlebot and Bingbot as examples, so keep that category out of any block rule. If Googlebot already gets a 403, how to find which layer blocked it is on its own page.

Which settings should I use for my goal?

Your goalTrainingSearchAgentNote
Keep content out of model training, stay listed in AI searchBlockAllowAllowThe most common choice for publishers
Leave AI search too, keep GoogleBlockBlockAllow or blockCheck that Googlebot is untouched
Store with prices and stock you do not want read by agentsBlockAllowBlockAgents are the hardest to identify
Docs or marketing site that wants AI citationsAllowAllowAllowWatch load in your analytics
Pages behind login or with customer dataBlockBlockBlockProtect these with authentication; bot rules are a second layer

The other AI crawler guides go deeper on each bot. Allowing Search is what keeps search bots such as OAI-SearchBot able to read your pages. If you turn on a blanket block and then drop out of AI answers, check your own Search policy first.

Can robots.txt do the job instead?

For well-behaved crawlers it is the polite option, and Anthropic's guidance for site owners is to use robots.txt and not IP blocking, because blocking IPs can stop the crawler reading your opt-out. OpenAI says changes can take about 24 hours to take effect for its systems.

It enforces nothing. Cloudflare's managed robots.txt page says compliance is voluntary and that for enforcement you need AI Crawl Control. RFC 9309 says the same in its own words: the rules are not a form of access authorization.

Cloudflare's managed robots.txt can write the directives for you from your policies. Turn on Bot Preference Sync after setting Training to Disallow. It adds Content Signals and Disallow lines for known AI bots such as ClaudeBot and GPTBot.

Google-Extended is a special case, and llms.txt restricts nothing either. Google says it has no separate user agent string and does not affect inclusion in Search or ranking. It is a robots.txt token only, so a WAF user agent rule cannot enforce it.

How are agents different, and what is Web Bot Auth?

An agent acts for one person at a time, so you cannot reason about it as a crawler. Google's own agent has its own verify-or-block page, and shop owners can read whether a store should let shopping agents in. Cloudflare's signed agents page says signed agents became Verified on 1 July 2026 and separates Direct bots (one operator) from Intermediary ones (a service many users can drive).

Web Bot Auth is the technical basis: a bot signs its requests with a key and publishes the public key, and Cloudflare verifies the signature. Not every agent signs, so unsigned agents fall back to user agent strings and IP lists. Perplexity, for example, publishes IP lists for both its agents and recommends combining user agent matching with IP verification.

A rule that says "block agents" therefore covers the agents Cloudflare can identify. It does not cover a browser automation tool that looks like a person.

How do I test the rules and avoid blocking Google?

Test before you trust a rule, and test the opposite direction too.

  • Send a request from your own machine with a crawler user agent and confirm a block or challenge response. Use a site you own.
  • Open Security Events and filter by the rule name. Confirm the matches are the bots you expected.
  • Use the URL Inspection live test in Google Search Console and fetch your sitemap, to confirm Googlebot still gets a 200.
  • If you are on the Free plan and see unexplained blocks, check Bot Fight Mode. Cloudflare states it cannot be bypassed or skipped with WAF custom rules, and that it can challenge API or mobile app traffic.
  • Re-check after 24 hours for OpenAI and again after a week for traffic in your analytics.

A user agent test proves your rule matches that string. It does not prove a real crawler is blocked, because real crawlers arrive from their own addresses and are identified by verification.

What will a firewall rule not do?

It does not remove content that has already been collected, and it does not make private pages private. If a page should not be read by anyone outside your team, it needs authentication.

A bot rule also does not hide your origin. If your server answers requests sent straight to its IP address, a crawler that finds it never meets Cloudflare. Cloudflare does not shield a directly reachable origin server by default, and a WAF is a stopgap, not a patch. Crawlers that ask for sensitive files are a different problem; see bots requesting .env and .git.

An outside scan can read what your site serves. It cannot see your Cloudflare rules or tell you whether a bot would be blocked; the guide to what outside scans miss explains the limit.

Questions people ask

How do I block all AI crawlers in Cloudflare?

Open Security Settings, then Configure AI bot policies, and set Search, Training and Agent to block. That covers verified AI bots and some unverified ones. Add a custom rule on user agents for anything the policy misses, and test with a request that sends a crawler's user agent.

If I block AI bots, will Google still index my site?

Yes, as long as Googlebot is not blocked. Google says its Google-Extended token does not affect inclusion in Google Search or ranking. Be careful with the Search policy and with category rules, because a rule that is too wide can catch search crawlers as well.

Does robots.txt stop ChatGPT and Perplexity from reading my pages?

It stops the crawlers that choose to follow it. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them. Those fetches are triggered by a person. Only a firewall rule can refuse them.

Why is my site invisible in AI answers after I turned on Cloudflare's AI block?

The Search policy or a category rule may be blocking the bots that build AI search results, such as OAI-SearchBot. Blocking training crawlers and allowing search crawlers are separate settings. Check the policy for each behaviour and the Security Events log for blocked AI bots.

Can I block AI agents by user agent alone?

You can, but a user agent string is easy to fake in both directions. Cloudflare verifies bots through signatures, published IP lists or reverse DNS. Use user agent rules as a second layer behind the verified-bot policies, and do not treat them as proof of who sent a request.

Sources

  1. Cloudflare: Block AI botsDashboard path, the Search, Training and Agent policies, the legacy toggle's gap, the 15 September 2026 default change. Checked 2 November 2026.
  2. Cloudflare: AI Crawl Control overviewPer-crawler allow or block rules, available on all plans. Page updated 14 August 2026.
  3. Cloudflare: Managed robots.txtSays robots.txt compliance is voluntary.
  4. Cloudflare: Verified botsHow a bot becomes verified; Search, Agent and Training behaviours; Direct and Intermediary.
  5. Cloudflare: Verified bot categoriesThe legacy cf.verified_bot_category values and the July 2026 consolidation of AI Search.
  6. Cloudflare: Signed agentsSigned agents became Verified on 1 July 2026.
  7. Cloudflare: Web Bot Auth
  8. Cloudflare: WAF custom rulesActions and per-plan rule limits.
  9. Cloudflare: Bot Fight ModeStates that Bot Fight Mode cannot be skipped with custom rules.
  10. OpenAI: Overview of OpenAI crawlersOAI-SearchBot, GPTBot, ChatGPT-User and OAI-AdsBot, and which follow robots.txt.
  11. Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?
  12. Perplexity: Perplexity crawlers
  13. Google: Common crawlers (Google-Extended)
  14. RFC 9309: Robots Exclusion ProtocolSection 1.3: the rules are not a form of access authorization.

Facts on this page are dated and point at the primary source next to them. If a vendor status changes after the date shown, the vendor's page wins; tell us at support@run.cloud and we will update it.