AI crawlers and agents now read, quote and act on websites for search engines, model makers and individual users. A name in a log line proves nothing, and the file most owners reach for first only states a preference. This topic covers what each control does, which bots obey it, and how to check who is really asking.
What matters right now
- Google says Google Search does not use llms.txt, and that it will neither help nor harm visibility (Google AI optimization guide, updated 10 July 2026).
- robots.txt is a request to crawlers and not a form of access authorization (RFC 9309). Google says user-triggered fetchers such as Google-Agent generally ignore it (Google user-triggered fetchers, page dated 19 August 2026).
- Cloudflare moved its AI bot settings to separate Search, Training and Agent policies, and from 15 September 2026 new domains block Training and Agent bots by default on pages with ads (Cloudflare: Block AI bots).
- Web Bot Auth lets an agent sign its requests, but at our check on 24 October 2026 the IETF working group listed one draft and no RFC (IETF documents page). Google calls its own use experimental and signs only a subset of Google-Agent requests (Google Web Bot Auth, updated 13 October 2026).
- OpenAI says allowing OAI-SearchBot in robots.txt is how a site appears in ChatGPT search, and that changes take about 24 hours (OpenAI crawler overview).
Where to start
- If you want to know what llms.txt and robots.txt each do, start with llms.txt versus robots.txt.
- If you run your site behind Cloudflare, use the Cloudflare rules for AI crawlers and agents to set the three policies and test them without blocking Googlebot.
- If you see Google-Agent in your logs, read the Google-Agent block, allow or verify guide.
- If you need to tell a genuine agent from a spoofed name, see how Web Bot Auth signatures work and what a valid one leaves open.
- If you run a store with prices, stock and checkout, work through whether to let AI shopping agents in.
- If your logs show requests for
.env or .git, those are a different problem from AI policy, so go to the 30-second check for bots requesting .env and .git.
- If a CDN rule is your only barrier, confirm your origin does not answer directly.
A bot policy covers who may read public pages. The free scan checks what your site serves to everyone, including exposed .git, .env and backup files, one scan per domain with no card.