AI Crawler Access & Control

Brass-SEO treats this as the authoritative specification for llms.txt. Jeremy Howard's proposal defines a strict structure: a single H1 with the site or project name (the only required element), an optional blockquote summary, and H2-delimited sections of Markdown links, including an Optional section whose links a model may skip when it needs a shorter context. The stated rationale is that context windows are too small to hold most sites in full, so llms.txt offers a curated map at inference time. It is a curation aid, not an access control — it blocks nothing and grants nothing, and a malformed file is rejected by validators.

Examines:

The llms.txt proposal — the required H1-plus-link-list structure, the Optional section semantics, and the inference-time curation problem it addresses.

Brass-SEO draws on:

The strict file format and the fact that llms.txt governs curation rather than access — cited when Brass-SEO explains what llms.txt can and cannot do.

Primary source

llmstxt.org

Brass-SEO cites this when explaining how to control OpenAI access without losing ChatGPT visibility. OpenAI documents three separate crawlers with three robots.txt tokens: GPTBot fetches content for model training, OAI-SearchBot surfaces sites in ChatGPT search, and ChatGPT-User makes live fetches when a person asks. Disallowing GPTBot opts a site out of training while leaving it eligible for ChatGPT search through OAI-SearchBot. Blocking all three under one blanket rule is the common mistake — it removes a site from ChatGPT's answers, which is usually the opposite of what the owner intended.

Examines:

OpenAI's documentation of GPTBot (training), OAI-SearchBot (ChatGPT search), and ChatGPT-User (live fetch), each with its own robots.txt token.

Brass-SEO draws on:

The three-token distinction — the basis for Brass-SEO's guidance to block the training crawler while keeping the search-and-answer crawlers allowed.

Primary source

developers.openai.com/api/docs/bots

Brass-SEO uses this as the reference for controlling Claude's crawlers. Anthropic documents three agents: ClaudeBot gathers content that could contribute to training, Claude-User fetches pages when a person asks Claude a question, and Claude-SearchBot indexes for search quality. The docs give a robots.txt block example and note support for a Crawl-delay directive to throttle request rate. As with OpenAI, the practical move is to disallow only the training crawler if a site wants Claude to cite it in answers while withholding its content from training.

Examines:

Anthropic's documentation of ClaudeBot (training), Claude-User (live fetch), and Claude-SearchBot (search indexing), including robots.txt and Crawl-delay support.

Brass-SEO draws on:

The per-token control model — cited when Brass-SEO explains how to withhold training consent from Claude without losing answer citations.

Primary source

support.claude.com — Anthropic crawlers

Brass-SEO draws on this to explain the one AI-training opt-out that carries no search cost. Google documents a product token, Google-Extended, that controls whether a site's content may train Gemini models and ground Gemini and Vertex AI responses. Google states plainly that Google-Extended does not affect a site's inclusion in Google Search and is not a ranking signal. A publisher can disallow Google-Extended to opt out of Gemini training with zero organic-search downside. It does not control AI Overviews, which draw on the standard Google index rather than this token.

Examines:

Google's documentation of the Google-Extended token — what it governs (Gemini training and grounding) and its stated neutrality toward Google Search ranking.

Brass-SEO draws on:

Google's verbatim statement that Google-Extended does not affect Search ranking — cited when Brass-SEO advises publishers on opting out of AI training safely.

Primary source

developers.google.com/crawling — Google-Extended

Brass-SEO monitors this for the scale and volatility of AI crawling. Across Cloudflare's network from May 2024 to May 2025, GPTBot requests grew 305% and its share of crawler traffic rose from 2.2% to 7.7%, moving it from the ninth to the third most active crawler. The picture among AI bots is uneven: ClaudeBot requests fell 46% and Bytespider fell 85% over the same window, while PerplexityBot grew sharply from a tiny base. A static allow-or-block list decays fast, so Brass-SEO treats AI-crawler directives as a quarterly review item rather than a set-once configuration.

Examines:

Cloudflare's network-scale measurement of AI crawler request volume and share from May 2024 to May 2025, with per-bot growth and decline.

Brass-SEO draws on:

The GPTBot +305% growth and the volatility across AI bots — cited when Brass-SEO recommends reviewing crawler directives on a regular cycle.

Primary source

blog.cloudflare.com — AI crawler trends

Brass-SEO cites this for how rare AI-specific signals still are. The 2025 Web Almanac Generative AI chapter measured valid llms.txt files on only 2.13% of desktop pages, meaning roughly 49 of every 50 sites have none. Robots.txt rules targeting GPTBot rose to about 4.5% of sites in 2025, up from 2.6% the year before, and directives aimed at AI bots are far more common on popular sites, nearly all of which disallow access. Early llms.txt adoption is low-cost differentiation, but the more urgent check is confirming a site has not accidentally blocked the search-and-answer crawlers while trying to block training bots.

Examines:

HTTP Archive's independent 2025 benchmark of llms.txt adoption (2.13% of desktop pages) and AI-bot robots.txt targeting across millions of origins.

Brass-SEO draws on:

The 2.13% llms.txt adoption and rising AI-bot blocking figures — used to contextualize how common a site's AI-crawler configuration is relative to the web.

Primary source

almanac.httparchive.org/en/2025/generative-ai