llms.txt: A Practical Guide for AI Crawlers

The honest answer first: llms.txt is a proposed standard for guiding AI crawlers, not a confirmed ranking or citation factor. Major AI platforms — OpenAI, Anthropic, Google, Perplexity — have not publicly committed to reading llms.txt files. The standard exists, adoption is growing, and the cost of implementing it is low, but the citation impact is unproven.

This post covers what llms.txt is, how to write one, what we know about adoption, and how Brass-SEO implements it. If you are looking for a guaranteed AI traffic boost, llms.txt is not it. If you are looking for a low-effort hedge that may help and cannot hurt, it is reasonable.

For the full GEO framework, see the Generative Engine Optimization guide.

What llms.txt Is
The Current State of Adoption
The File Format and Conventions
How to Write llms.txt for Your Site
llms-full.txt and the Two-File Pattern
How Brass-SEO Uses llms.txt
What llms.txt Cannot Do
Frequently Asked Questions

What llms.txt Is

llms.txt is a proposed standard for a markdown file at the root of your website that gives AI systems a curated, machine-readable summary of your most important content.

The proposal was published in September 2024 by Jeremy Howard, co-founder of Answer.AI and creator of fast.ai. The specification lives at llmstxt.org. The idea is straightforward: large language models have limited context windows and need to process lots of pages quickly. A single curated file at /llms.txt lets a site point AI systems at its most important content directly, instead of forcing them to crawl, render, and synthesize the entire site.

The format is markdown. It uses a defined structure — H1 for the site name, blockquote for the elevator-pitch summary, H2 sections that group links by topic — that makes it both human-readable and easy for AI to parse. A page-level companion file, /llms-full.txt, can contain the full text of your most important pages for AI systems that want everything in one fetch.

This is similar in spirit to robots.txt and sitemap.xml — a site-level signal that gives crawlers structured guidance — but specifically targeted at AI consumers rather than search engine spiders.

The Current State of Adoption

llms.txt adoption among publishers is growing but no major AI vendor has publicly confirmed they read or weight llms.txt during inference or training.

What is known publicly: adoption is visible across SaaS companies, documentation platforms, and developer tools. The llmstxt.org site maintains a directory of adopters. The standard is widely discussed in AI engineering circles, and tooling for generating llms.txt files exists in several frameworks.

What is not known publicly: whether ChatGPT, Perplexity, Claude, or Google AI Overviews give llms.txt any weight in their citation or training behavior. None of the major AI vendors have published statements confirming or denying that they read it. Independent research measuring llms.txt's effect on AI citations has not been published as of this writing.

The asymmetry matters. The cost of implementing llms.txt is one or two hours of work and a single static file. The upside is unknown but plausibly positive for any AI system that does parse it. The downside is essentially zero. That math is why adoption is growing despite the unconfirmed payoff.

The File Format and Conventions

llms.txt follows a structured markdown format defined in the llmstxt.org specification: site name as H1, elevator-pitch summary as a blockquote, H2 sections grouping curated links.

The convention is:

# Site Name

> One-paragraph summary of what this site is and what it offers.

Optional supporting paragraph that adds context the summary cannot fit.

## Key Section Name

- [Page Title](https://example.com/page): Brief description
- [Another Page](https://example.com/other): Brief description

## Another Section

- [Important Doc](https://example.com/doc): Brief description

Each H2 should group thematically related pages. Each link should be a fully qualified URL pointing to important content on your site. Each description should be a short summary AI can read without fetching the linked page.

The file lives at /llms.txt (so https://yourdomain.com/llms.txt). It must be served as plain text or markdown — not as HTML.

How to Write llms.txt for Your Site

The most useful llms.txt is a curated index of your highest-value pages, not an exhaustive list. Pick 15-30 pages that best represent what your site is about and what it teaches.

Start with this structure:

Site name (H1). Your business or site name.
Elevator pitch (blockquote). One paragraph that explains what your site is, who it serves, and what it offers. Make it specific. "AI-powered SEO analysis for small business — connect Google Search Console and GA4, chat with AI for plain-English insights" is more useful than "An SEO platform."
Context paragraph (optional). A second paragraph that adds detail the summary cannot fit. Pricing, scope, or unique value proposition.
Section: Key Features or Services. Use bullet points with bold feature names and one-line descriptions.
Section: Essential Guides. Link to your pillar content, comprehensive guides, and frequently referenced pages.
Section: Use Cases or Customer Types. Describe who the site serves and what they typically come to do.
Section: Pricing and Trial. If applicable, state pricing clearly.
Section: Support and Policies. Link to support, terms, privacy.

Keep each link description under 15 words. Avoid marketing fluff. Treat it like writing for an analyst who has 30 seconds to understand your site.

For sites with deep documentation, organize H2 sections by user goal — "Getting Started," "Reference," "Tutorials," "Integration" — rather than by site navigation. AI systems use llms.txt to figure out which page best answers a question, so the grouping should match the question someone might ask.

llms-full.txt and the Two-File Pattern

A companion file at /llms-full.txt provides the full text content of your highest-priority pages, concatenated into a single file. Some AI systems fetch this for one-shot context retrieval.

The two-file pattern works like this:

/llms.txt is the index. It is short, curated, and points to where to read more.
/llms-full.txt is the contents. It includes the actual text of your most important pages, concatenated, so an AI system can ingest your site's substance in a single request.

Implementing llms-full.txt is more work — you have to maintain it as content changes — but for documentation-heavy sites or sites where AI customer support is a real use case, it is worth doing. Most marketing sites do fine with just llms.txt.

If you are using a static site generator or a CMS with a markdown export, you can often automate llms-full.txt generation as part of your build pipeline. Manual maintenance is sustainable for sites with under 50 priority pages.

How Brass-SEO Uses llms.txt

Brass-SEO maintains an llms.txt file at brass-seo.com/llms.txt that lists the service overview, pricing, key guides, and support pages. The GEO guide page also embeds an inline llms.txt-style summary.

The site-level file follows the standard structure: H1 for "Brass-SEO," a blockquote summary describing the service, sections for Key Features, Pricing, How It Works, Use Cases, Technical Specifications, AI Prompts Repository, Essential Guides, and Support.

The page-level pattern is different. The Brass-SEO Generative Engine Optimization guide includes an inline llms.txt-style summary inside the page itself, as a <script type="text/llms.txt"> block. This is not part of the llmstxt.org specification — it is an experimental pattern for embedding page-specific guidance for AI systems that parse the rendered HTML. Whether AI systems read this inline format is unconfirmed.

We are not claiming the Brass-SEO llms.txt has produced measurable AI citation gains. We have it because the cost is trivial and the standard is plausible. This is the same logic any small business should apply.

What llms.txt Cannot Do

llms.txt cannot rescue thin content, force AI systems to cite you, or replace structural improvements like answer capsules and original data.

The dominant citability factors identified in research are content patterns, not metadata. The Princeton GEO study (Aggarwal et al., 2024) found expert quotes drove a +41% citation lift. The Gnuse 2025 practitioner study found 72.4% of cited posts contained answer capsules. None of those gains come from llms.txt — they come from how the page itself is written.

llms.txt also cannot substitute for traditional crawler signals. AI systems that source content via search engines (Google AI Overviews) or that follow standard web crawls (most retrieval-augmented generation systems) need your site to be indexable, performant, and well-linked. llms.txt is supplementary, not a replacement for the basics.

Finally, llms.txt cannot distinguish your content from your competitors' content unless your content actually has something distinguishable in it. The file points AI to your pages. The pages still have to earn the citation.

Frequently Asked Questions

Do I need an llms.txt file?

No, but you may benefit from one. The file is a low-cost hedge: 1-2 hours to create, no maintenance for static sites, no downside if AI systems ignore it. If you are doing other GEO work, adding llms.txt is a reasonable small addition. If you are just starting on AI search visibility, focus on answer capsules and content structure first.

Will llms.txt improve my AI search rankings?

There is no public evidence that it does or does not. Major AI vendors have not confirmed they read llms.txt. Independent citation research has not isolated llms.txt as a variable. Treat any claim of measured citation gains from llms.txt as unverified until research catches up.

Where should I host llms.txt?

At the root of your site, served as plain text or markdown. The URL must be https://yourdomain.com/llms.txt. Most static hosting platforms (Vercel, Netlify, Cloudflare Pages) allow you to drop a file in your /public directory and have it served at the root automatically.

What is the difference between llms.txt and robots.txt?

robots.txt tells crawlers what they can and cannot access. llms.txt tells AI systems what is most important to read. They serve different purposes and do not replace each other. Most sites that have llms.txt also have robots.txt — they are complementary signals.

Should I block AI crawlers with robots.txt instead of using llms.txt?

That is a separate decision driven by your stance on AI training. If you want AI systems to use your content (typical for businesses that want AI citations), do not block them in robots.txt — and consider adding llms.txt to make their job easier. If you do not want AI systems to use your content, robots.txt is the right tool, and llms.txt is irrelevant.

How often should I update llms.txt?

When your site's main pages or core offerings change. For most small business sites, this is once or twice a year — when you launch a new service, change pricing, or publish a major guide. The file is a curated overview, not a real-time index, so it does not need to update with every blog post.

Does Brass-SEO have an llms.txt?

Yes, at brass-seo.com/llms.txt. It lists the service overview, pricing, key features, and links to essential guides. We have not measured a citation lift from it specifically — it is part of a broader GEO strategy that also includes answer capsules, original research, and named expert citations.