1. Red
  2. Learn
  3. AI crawlers and bots

Google-Agent user agent: block, allow or verify it?

Google-Agent acts for a user and generally ignores robots.txt. See its user agent, how to verify it by IP range or signature, and when to block it.

By Timothy Lindblom, FounderPublished 4 November 20267 min readHow we check facts
On this page
  1. What is Google-Agent, and is it Googlebot?
  2. Does robots.txt stop Google-Agent?
  3. Why can't I trust the user agent string?
  4. What is Web Bot Auth, and should I verify signatures?
  5. Block, allow or verify: how do I decide?
  6. What should I ask my host or CDN?
  7. What can a scan tell me about this?
  8. What do we not know yet?

Google-Agent is a Google fetcher that visits your site when a person asks an agent on Google infrastructure to browse or act. Google says such fetchers generally ignore robots.txt, so a Disallow line is not a block. Decide per path, then verify each request by Google's published IP ranges or by its Web Bot Auth signature before trusting the name.

FactDetailSource
What it isAn agent on Google infrastructure that navigates the web and performs actions on a user's requestGoogle-Agent page, dated 2026-08-19
robots.txt"Generally ignore robots.txt rules", because a user asked for the fetchUser-triggered fetchers, dated 2026-08-19
IP rangesuser-triggered-agents.json, 20 prefixes, generated 2026-11-03Google JSON file
SignaturesWeb Bot Auth, experimental; a subset of requests is signedWeb Bot Auth page, dated 2026-10-13
Checked4 November 2026this page

What is Google-Agent, and is it Googlebot?

No, it is a different thing. Google sorts its fetchers into three categories: common crawlers such as Googlebot, which always respect robots.txt for automatic crawls; special-case crawlers; and user-triggered fetchers, where an end user triggers the fetch. Google-Agent is in the third group, next to fetchers such as Google Read Aloud and Google Site Verifier on Google's list.

The practical difference is who decides. Googlebot decides by itself what to crawl. Google-Agent arrives because a person told an agent to go to your page and do something. That could be reading a price, filling a form or comparing products. Your server sees a browser-like visitor with a Google label.

The user agent strings are in Google's documentation. Both a mobile and a desktop string exist. Each is a Chrome-style string that contains compatible; Google-Agent; and a link to that documentation page. Search your logs for the token Google-Agent, not for the whole string, because the Chrome version part changes.

Does robots.txt stop Google-Agent?

Treat it as no. Google's page says that because the fetch was requested by a user, these fetchers generally ignore robots.txt rules. At our check on 4 November 2026 the Google-Agent page described no robots.txt token for it. The word "generally" is Google's, so behaviour may change, and you cannot rely on either outcome.

A User-agent: Google-Agent group with Disallow: / is harmless, and it asks politely for something Google has not promised. Do not count it as a control.

This is not special to Google. Google's robots.txt introduction says the instructions in a robots.txt file cannot enforce crawler behaviour. The IETF standard, RFC 9309, says robots.txt rules are not a form of access authorization. A file that must not be read needs a login, a firewall rule or removal from the web root.

One related token is easy to confuse with this one. Google-Extended is a robots.txt product token that manages whether crawled content may be used to train Gemini models. Google says it has no separate request user agent. It controls a use of content already crawled, and it does nothing about Google-Agent visits. An llms.txt file is no control either, as what actually limits AI access explains, and the other AI crawler guides cover the remaining bots.

Why can't I trust the user agent string?

Any client can send any string. A scraper that wants to look friendly can write Google-Agent into its headers in one line. The name tells you which label the sender chose, nothing about who sent it.

Google gives you two ways to check, and they answer different questions.

CheckWhat it provesWhat it cannot proveSource
Source IP in user-triggered-agents.jsonThe request came from an address Google lists for these agentsWhich user asked, or what they intendGoogle IP file
Reverse DNSFor user-triggered fetchers Google lists gae.googleusercontent.com and google-proxy namesWhether the Google-Agent file ranges use these names; the page does not sayVerify Google requests
Web Bot Auth signatureThe request was signed with a key published at agent.bot.googAnything about unsigned requests, which are legitimate tooGoogle Web Bot Auth

Two details matter here. First, Google's verification page lists user-triggered-agents.json as its own file, separate from user-triggered-fetchers.json. Check the right file. Second, the file changes. It was regenerated on 3 November 2026 at our check, so fetch it on a schedule instead of pasting the ranges into a rule once.

What is Web Bot Auth, and should I verify signatures?

Web Bot Auth is, in Google's words, an experimental cryptographic protocol for authenticating requests from bots. Google says it is testing it with some AI agents on its infrastructure, that not all Google user agents use it, and that only a subset of Google-Agent requests are signed.

A signed request carries a Signature-Agent header naming https://agent.bot.goog, plus Signature and Signature-Input headers. Google tells sites to fetch the public keys from https://agent.bot.goog/.well-known/http-message-signatures-directory, check the signature as described in RFC 9421, and fall back to IP-based checks, because not all requests are signed. At our check that directory listed five Ed25519 public keys.

The consequence for your rules is a three-way split:

  • A valid signature plus an expected IP is the strongest evidence you can get.
  • An unsigned request from a listed IP is probably genuine. Google says unsigned requests exist.
  • A request that claims the name, comes from neither a listed IP nor carries a valid signature, is not Google-Agent whatever its header says.

You rarely need to write the verification yourself, and how site owners check a signed agent request is covered step by step on its own page. Cloudflare documents how it validates Web Bot Auth signatures against a key directory, and its signed agents programme, launched 28 August 2025, classifies agents that are directed by an end user. If you already sit behind a CDN or WAF, look at what it reports for this agent before building anything, and see the Cloudflare rules for AI agents if that is your edge. The origin IP caveat applies: rules at the edge do not cover requests that reach your server directly.

Block, allow or verify: how do I decide?

Start from what a visit can do on each part of your site. An agent acting for a user behaves like that user, minus any login you did not give it. If you run a shop, the decision for AI shopping agents goes further.

Path typeSuggested stanceReason
Public articles and product pagesAllowA person asked to read the page, which is the same as a visitor
Search, filters and forms that write dataRate limit, verify before allowingAn agent can submit forms faster than a person
Login, checkout and account pagesBlock or require the visitor to sign inNothing about an agent visit proves who is acting
Admin panels, staging, internal toolsBlock everyone not on an allow listThese should never rely on a bot policy
Files that must stay private, such as .env, .git, backupsRemove from the web root and rotate anything exposedNo bot rule is a control for these

The last row is the one that matters most. A file that is publicly readable is readable by every client, whichever name it uses. If you suspect that, read bots requesting .env and .git and is your .env file publicly readable. The free scan checks .git, .env, backups and open directory listings, and reports what the site serves.

How do I block it at the server?

Pick the rule by how sure you need to be. A user agent match stops honest clients that use the name and is trivially bypassed. An IP-range rule needs the Google file kept current. A rule that requires a valid signature is the strongest, and it blocks unsigned genuine requests too, since Google says only some are signed.

A minimal user agent rule in nginx looks like the example below. Adapt it to your setup and test it on a staging server first, since a typo in a server block can take a site down.

if ($http_user_agent ~* "Google-Agent") { return 403; }

Send the denial only on the paths you decided on, and log the matches for a week. Then compare the logged source IPs against the Google file. Matches from outside it are impostors that you can block under any name.

  • Search access logs for the token Google-Agent for the last 30 days.
  • Compare source IPs with user-triggered-agents.json, fetched today.
  • Decide allow, rate limit or block for each row of the table above.
  • Put the block on login, checkout and admin paths first.
  • Keep Googlebot rules separate, and watch Search Console after any change.
  • Re-fetch the IP file on a schedule.

What should I ask my host or CDN?

If you do not control the server, send one question. Hosts and CDNs differ in what they log, so ask for facts.

Does your platform identify Google-Agent requests, and how? Does it check the source IP against Google's user-triggered-agents.json, or verify Web Bot Auth signatures from agent.bot.goog? Can I write a rule for this agent on my login and checkout paths, and can I see the matching requests in my logs?

An answer that names the IP file or the signature check is an answer you can use. An answer that says "we block bad bots" tells you nothing about this agent.

What can a scan tell me about this?

A scan from outside cannot see who visited you. It reads what your site serves. That is useful on this topic in one narrow way: it can tell you whether files and pages that no bot should ever reach are being served to everyone. The free scan reads the stack your site serves and matches each version to known vulnerabilities, with checks for exposed .git, .env and backup files. It is a lookup, and what it cannot see is covered in the limits of an outside-in check.

If a log search turns up requests to paths you never meant to publish, whatever name they used, follow the step-by-step response plan.

What do we not know yet?

Google's pages leave several questions open at our check on 4 November 2026. They do not say how much of Google-Agent traffic is signed, whether it will follow robots.txt in future, or whether the reverse DNS names for user-triggered fetchers apply to the ranges in the agents file. Read the dated pages linked in the sources again before relying on any of those, since the Web Bot Auth page was updated on 13 October 2026 and the agents IP file the day before this check.

Questions people ask

How do I block Google-Agent in robots.txt?

Google's page for Google-Agent says user-triggered fetchers generally ignore robots.txt rules, and the page we read lists no robots.txt token for it. A Disallow line is a request that this fetcher is not documented to honour. To refuse it, block at the web server, CDN or firewall using the signals described on this page.

What is the Google-Agent user agent string?

Google documents a mobile and a desktop string. Both contain the text compatible; Google-Agent followed by a link to Google's Google-Agent documentation page, inside a Chrome-style string. The string alone proves nothing, because any client can send it.

Is Google-Agent the same as Googlebot?

No. Googlebot is a common crawler that always respects robots.txt for automatic crawls. Google-Agent is a user-triggered fetcher that acts for a person who asked an agent on Google infrastructure to do something. Blocking one does not affect the other.

Will blocking Google-Agent hurt my Google search ranking?

Google's pages describe Google-Agent as a separate fetcher from Googlebot and do not link it to Search indexing. Do not block Googlebot ranges or the Googlebot user agent while doing this. Keep the rules for the two completely separate, and watch Search Console after any change.

How do I know a request really came from Google-Agent?

Check the source IP against Google's user-triggered-agents.json file. Some requests also carry a Web Bot Auth signature that you can verify against the keys at agent.bot.goog. Google says only a subset is signed, so an unsigned request from a listed IP can still be genuine.

Should I allow Google-Agent on my site?

It depends on what a person could ask it to do on your pages. Allow it on public content where an agent visit equals a visitor. Rate limit or block it on login, checkout and admin paths, and treat it as an unauthenticated visitor either way.

Sources

  1. Google-Agent (Google Search Central, crawlers and fetchers)User agent strings, robots.txt behaviour, IP range file. Page dated 2026-08-19. Checked 4 November 2026.
  2. Google user-triggered fetchersSays these fetchers generally ignore robots.txt; lists Google-Agent. Page dated 2026-08-19. Checked 24 October 2026.
  3. Web Bot Auth (Google Search Central)Experimental; a subset of Google-Agent requests is signed; verify steps. Page dated 2026-10-13. Checked 4 November 2026.
  4. Verifying requests from Google's crawlers and fetchersReverse DNS domains and the IP range JSON files per crawler type. Page dated 2026-03-20. Checked 24 October 2026.
  5. user-triggered-agents.json (Google IP ranges)20 prefixes, creationTime 2026-11-03T15:45:48 at our check on 4 November 2026.
  6. agent.bot.goog key directory (Web Bot Auth)Five Ed25519 public keys listed at our check on 4 November 2026.
  7. Overview of Google crawlers and fetchersThe three categories and which ones always respect robots.txt. Page dated 2026-06-12.
  8. Google common crawlers, including Google-ExtendedGoogle-Extended is a robots.txt token with no separate request user agent. Page dated 2026-10-15.
  9. Introduction to robots.txt (Google Search Central)Google says robots.txt cannot enforce crawler behaviour. Page dated 2025-12-10.
  10. RFC 9309: Robots Exclusion Protocol (IETF, 2022)States that robots.txt rules are not a form of access authorization.
  11. RFC 9421: HTTP Message Signatures (IETF, 2024)The signature standard Google's Web Bot Auth page tells sites to verify against.
  12. Cloudflare: Web Bot AuthThe three headers and how Cloudflare checks a signature. Page dated 2026-10-08.
  13. Cloudflare: signed agents (28 August 2025)Defines signed agents as agents directed by an end user; first cohort.

Facts on this page are dated and point at the primary source next to them. If a vendor status changes after the date shown, the vendor's page wins; tell us at support@run.cloud and we will update it.