The Research Behind Blocking AI Crawlers
A business owner blocks GPTBot, checks ChatGPT a week later, and finds their competitor still gets recommended and they still don't. The robots.txt rule worked exactly as written. It just didn't do what they thought it would do.
Brass-SEO gets a version of this question often enough that it's worth going past the robots.txt mechanics and into the research underneath them: what a language model actually memorizes, why some brands get recalled reliably and others never do, and where the raw material for that memory comes from in the first place. The mechanics are already covered in Block the Training Bot, Keep the Answer Bot. This post is the evidence layer underneath it.
Quick Navigation
- Two Different Bots, Two Different Jobs
- What a Language Model Actually Memorizes
- Why Some Brands Get Remembered and Others Don't
- Where the Raw Material Comes From
- What Each Company Actually Discloses
- What This Changes About Your robots.txt
- Frequently Asked Questions
Two Different Bots, Two Different Jobs
Brass-SEO treats "training" and "retrieval" as two separate systems that happen to share a company name. Training is what happened when a model like GPT-4 or Claude was built: a crawler read a portion of the web once, at some point in the past, and the model absorbed patterns and facts from that text into its weights. Retrieval is what happens right now, when you ask ChatGPT or Claude a question and it goes and reads a live page to answer it.
Those two processes are controlled by different crawlers, and blocking one doesn't touch the other. A brand already folded into a model during a past training run stays inside that model's memory regardless of what robots.txt says today. Understanding why requires looking at what "memory" means for a language model in the first place.
What a Language Model Actually Memorizes
A 2023 ICLR paper by Carlini and coauthors put a number on something practitioners had mostly guessed at: how much of a model's training text it can reproduce, word for word, given the right prompt. Testing the GPT-Neo and GPT-J model family, they found the 6-billion-parameter version memorizes at least 1% of its training dataset closely enough to reproduce it verbatim. That figure rises predictably as models get bigger — the researchers found a log-linear relationship, where roughly every tenfold increase in parameters adds about 19 percentage points of measured memorization, with a fit strong enough (R²=99.8%) that it functions like a rule rather than a trend.
Memorization also rises with how much surrounding context a prompt provides. At 50 tokens of preceding context, the researchers extracted 33% of tested training sequences; at 450 tokens, that jumped to 65%. A larger model, prompted with more context, recalls more of what it read. That's not a metaphor for how LLMs work. It's a measured property of the specific models tested.
The practical version: a business that shows up often, and in detail, across the pages a model trained on gets memorized, in the same technical sense the paper measured. That's why a well-documented brand gets named confidently and specifically, without the model needing to look anything up.
Why Some Brands Get Remembered and Others Don't
This is the part that explains the frustrating asymmetry — one competitor gets recommended, another with a comparable product never comes up. A 2023 ICML paper by Kandpal and coauthors tested this directly: they measured how a model's accuracy answering a factual question correlates with how many times the relevant fact appeared across the pretraining documents, using models up to 176 billion parameters on the TriviaQA benchmark.
The finding was blunt. Accuracy tracks document count closely, and for facts that show up in only a handful of pretraining documents, the researchers concluded current models would need to scale by many orders of magnitude before reaching competitive accuracy. Scale alone doesn't fix a document-count problem quickly. It takes a lot more scale than most people assume.
A business with a thin web presence isn't being penalized for quality when a model skips it. It's a document-count effect: the model encountered that business too rarely across training to build a reliable memory of it, the same way a person forgets a name they only heard once at a conference.
Where the Raw Material Comes From
The training text behind a model like GPT-3 comes overwhelmingly from the open web. The original GPT-3 paper by Brown and coauthors documents the model's exact pretraining mix: filtered Common Crawl supplied 410 billion tokens, weighted at 60% of the total training data — more than WebText2, two book corpora, and Wikipedia combined.
Common Crawl is a specific, ongoing project: a nonprofit that crawls the public web and publishes the results for free, running continuously since 2007. A page a business never wrote with an AI audience in mind can still end up training a future model, simply by being public and uncrawled-against. That's the mechanism behind "is my site in the training data" questions — see Is Your Site in the Data That Trained ChatGPT? for how to check.
What Each Company Actually Discloses
Each major AI company documents its crawlers separately by function, and the split matters more than the vendor names attached to it.
OpenAI names three crawlers with three jobs. GPTBot collects content for training future models and can be disallowed in robots.txt. OAI-SearchBot indexes pages for ChatGPT's search results. ChatGPT-User fires only when a live user's question triggers a fetch, and OpenAI states directly that robots.txt rules may not apply to those live requests.
Anthropic documents the same three-way split under different names. ClaudeBot collects content to train and improve models. Claude-User performs a live fetch on a real user's query. Claude-SearchBot crawls to improve search result quality. Each has its own independent robots.txt token, and Anthropic states it also honors the Crawl-delay extension for throttling request speed.
Google's version works differently. Google-Extended isn't a separate crawler with its own user agent — it's a control token layered onto Google's existing crawl, governing whether that content trains future Gemini models or feeds AI grounding. Google states explicitly that Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal." Disallowing it opts out of Gemini training without touching search visibility at all.
Common Crawl sits underneath much of this. Its crawler, CCBot, is independently blockable, and Common Crawl publishes its IP ranges so a site owner can verify a real CCBot request against a bot that's spoofing the user agent. Because Common Crawl feeds so many downstream training corpora at once, blocking CCBot is one of the few actions that reaches more than one company's models in a single robots.txt rule.
What This Changes About Your robots.txt
None of this changes the practical robots.txt setup Brass-SEO already recommends. It changes what a business owner should expect from it.
Blocking the four training crawlers covered above stops a business from contributing to future training runs. It does nothing to a brand already inside a model that finished training before the block went up, and it does nothing to what a chatbot retrieves live when a user asks it to browse or search. The live-retrieval crawlers are the ones a business generally wants left active, since those are the ones that can get it cited in a real answer.
The gap most business owners hit isn't a blocked crawler. It's a document-count problem: too few pages, mentioning the business too rarely, for any model to build a reliable memory of it. Brass-SEO's AI Citability audit checks a site's pages against the structural traits research associates with getting cited, which is the lever actually available to a business that wants to close that gap.
Frequently Asked Questions
Does blocking GPTBot remove my business from ChatGPT's answers?
No, not from what ChatGPT already learned. GPTBot controls whether OpenAI can use a page's content in a future training run; it has no effect on facts a past model already memorized, and it has no effect on ChatGPT-User's live browsing, which OpenAI states operates outside standard robots.txt enforcement.
Why does ChatGPT recommend my competitor but never mentions me?
The most likely explanation, based on Kandpal et al. (2023), is a document-count gap rather than a quality judgment. Their study found LLM accuracy on a fact tracks how often that fact appears across pretraining documents, and facts backed by only a few documents need far more model scale to answer reliably — scale most models don't have yet at that fact's document count.
Do language models really memorize specific facts, or just general writing patterns?
Both, and the memorization is measurable. Carlini et al. (2023) found a 6-billion-parameter model reproduces at least 1% of its training data verbatim on the right prompt, with memorization increasing predictably as model size grows. Larger models memorize more, not less.
If I want to stop a page from ever training a future AI model, what do I actually block?
Four separate tokens, since one blanket rule won't cover all of them: GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google), and CCBot (Common Crawl, which feeds many other companies' training corpora). Leave OAI-SearchBot, ChatGPT-User, Claude-User, and Claude-SearchBot unblocked if the goal is staying visible in live AI answers — see Block the Training Bot, Keep the Answer Bot for the exact robots.txt shape.
Full primary sources for every claim in this post, including the ones not fully spelled out above, are in the AI Training Data & LLM Memorization entry of the Brass-SEO Research Index — peer-reviewed papers and official crawler documentation, fetch-verified, no vendor studies.