Block the Training Bot, Keep the Answer Bot

You read a headline about AI stealing content, you feel a jolt, and you drop one blanket rule into your robots.txt to block AI. It feels responsible. It also just deleted you from the AI answers you wanted to win.

Here's the part nobody tells you. The bot that reads your pages to train a model and the bot that reads your pages to cite you in an answer are different bots with different names. Block them both with one line and you've locked the front door and the storefront at the same time. You wanted to stop the training. You stopped the sales.

Quick Navigation

• Why One Block Rule Backfires
• One Company, Several Crawlers
• A robots.txt That Opts Out of Training Only
• Google-Extended Is the Safe Lever
• Why llms.txt Is Not a Lock
• Frequently Asked Questions

Why One Block Rule Backfires

Brass-SEO sees this mistake constantly: a site owner writes one wildcard disallow to keep AI out and unknowingly removes the pages that would have shown up when someone asks ChatGPT a question. The blanket rule treats every AI crawler as the same threat. They aren't. Some read your site to feed a training run. Others read it to pull a live answer for a real person who is asking about your exact topic right now. That second group is traffic. That second group is where the buyers are.

Think of it like a warehouse and a shopfront under one company name. Shutting the warehouse doesn't have to shut the shop. You need to know which door is which before you bolt anything. Most people bolt both.

One Company, Several Crawlers

Brass-SEO tracks each AI company by its individual crawler tokens, because one company runs several bots that do different jobs. OpenAI runs three. GPTBot fetches content for model training. OAI-SearchBot surfaces your site inside ChatGPT search. ChatGPT-User does a live fetch the moment someone asks a question that touches your page. Disallow GPTBot and you opt out of training while OAI-SearchBot stays free to surface you in ChatGPT search. One name blocked, the other two working for you.

Anthropic follows the same pattern with three crawlers. ClaudeBot handles training. Claude-User does the live fetch. Claude-SearchBot handles search indexing. Anthropic also reads a Crawl-delay directive, so you can throttle how fast a bot requests pages instead of blocking it outright. The lesson repeats across vendors. Read the token names before you write a rule.

A robots.txt That Opts Out of Training Only

Brass-SEO recommends a targeted robots.txt that names each crawler instead of swinging at all of them. You allow the search bots that cite you and disallow the training bots that don't send anyone your way. Here's the shape of it:

User-agent: OAI-SearchBot
Disallow:

User-agent: Claude-SearchBot
Disallow:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

An empty Disallow means allow everything. A Disallow: / means block the whole site for that named bot. So the search crawlers get full access and the training crawlers get shut out. Adjust the names to match the vendors you care about. The structure holds.

Google-Extended Is the Safe Lever

Brass-SEO points to Google-Extended as the one AI-training opt-out that costs you nothing in search. It's a robots.txt token that controls whether your content trains Gemini models and grounds answers from Gemini and Vertex AI. Google states plainly that Google-Extended does not affect your inclusion in Google Search and is not a ranking signal. Blocking it does not push you down the results. You keep every bit of your search visibility and step out of the model training. That's a clean trade.

One caveat worth saying out loud. Google-Extended does not control AI Overviews. Those pull from Google's standard search index, the same index behind normal blue-link results. If you want out of AI Overviews, this token isn't the tool. It governs training and grounding. Nothing more.

Why llms.txt Is Not a Lock

Brass-SEO treats llms.txt as a menu, not a gate. Proposed by Jeremy Howard in 2024, it's a Markdown file that maps your key pages so a language model can read the important stuff at the moment it answers. It curates. It does not control access. It blocks nothing and grants nothing, which means it can't opt you out of training and can't keep any bot away from a single page. People confuse it with robots.txt all the time. They do opposite jobs.

Adoption is still tiny. As of 2025 only 2.13% of desktop pages carried a valid llms.txt file, per the HTTP Archive Web Almanac. Useful as a curation aid once your access rules are set. Useless as a fence.

Frequently Asked Questions

Will blocking GPTBot remove me from ChatGPT answers?

No. GPTBot only fetches content for model training. The bots that put you in ChatGPT are OAI-SearchBot for search and ChatGPT-User for the live fetch when someone asks a question. Block GPTBot and those two keep working. You opt out of training and stay eligible to be cited.

Does blocking Google-Extended hurt my Google rankings?

No. Google states plainly that Google-Extended does not affect inclusion in Google Search and is not a ranking signal. It only governs whether your content trains Gemini and grounds Gemini and Vertex AI answers. Your blue-link visibility stays exactly where it was.

Can llms.txt block AI crawlers?

No. llms.txt is a curation file that maps your key pages for models to read at inference time. It blocks nothing and grants nothing. Access control lives in robots.txt. Use llms.txt to guide what a model reads, not to keep anything out.

How often should I review my crawler rules?

Quarterly. The crawler mix shifts fast. Cloudflare network data shows GPTBot requests grew 305% from May 2024 to May 2025 and its share rose from 2.2% to 7.7%, moving from the 9th to the 3rd most active crawler, while ClaudeBot requests fell 46% over the same window. A rule that made sense in spring can be stale by fall.

Get the Full Crawler Map

The token names change and the traffic shares move every quarter, so this post sits on top of a living reference. The research index at /research/ai-crawler-control tracks each AI crawler, what it does, and how to allow or block it without wrecking your search visibility. Want to know which of these bots are already reaching your pages and which of your pages get cited? Start with a free first analysis and see where you actually stand.