1. Red
  2. Learn
  3. AI crawlers and bots

AI crawler guides: every article on bots and access

How to decide which AI crawlers and agents reach your site, what robots.txt and llms.txt control, and how to verify a bot. For site owners and store operators.

AI crawlers and agents now read, quote and act on websites for search engines, model makers and individual users. A name in a log line proves nothing, and the file most owners reach for first only states a preference. This topic covers what each control does, which bots obey it, and how to check who is really asking.

What matters right now

  • Google says Google Search does not use llms.txt, and that it will neither help nor harm visibility (Google AI optimization guide, updated 10 July 2026).
  • robots.txt is a request to crawlers and not a form of access authorization (RFC 9309). Google says user-triggered fetchers such as Google-Agent generally ignore it (Google user-triggered fetchers, page dated 19 August 2026).
  • Cloudflare moved its AI bot settings to separate Search, Training and Agent policies, and from 15 September 2026 new domains block Training and Agent bots by default on pages with ads (Cloudflare: Block AI bots).
  • Web Bot Auth lets an agent sign its requests, but at our check on 24 October 2026 the IETF working group listed one draft and no RFC (IETF documents page). Google calls its own use experimental and signs only a subset of Google-Agent requests (Google Web Bot Auth, updated 13 October 2026).
  • OpenAI says allowing OAI-SearchBot in robots.txt is how a site appears in ChatGPT search, and that changes take about 24 hours (OpenAI crawler overview).

Where to start

A bot policy covers who may read public pages. The free scan checks what your site serves to everyone, including exposed .git, .env and backup files, one scan per domain with no card.

Every ai crawlers and bots guide

2 articles, newest update first. Each one is dated and points at its primary sources.

Check your own site

The free scan reads the stack your site serves and matches each version to known vulnerabilities. Updated 4 November 2026.

Other topics