Crawling & Indexing

Brass-SEO treats this as the authoritative specification for sitemap implementation. The sitemaps.org protocol defines a 50,000 URL and 50 MB per-file hard limit, the four valid elements (loc, lastmod, changefreq, priority), and the sitemap index format for sites exceeding those limits. The lastmod field is the most actionable crawl-prioritization signal the spec provides — search engines read it to determine which URLs have changed and are worth re-crawling. Accurate lastmod values accelerate discovery of updated content; fabricated or static lastmod values train crawlers to ignore the field.

Examines:

The XML Sitemap protocol specification — format, element definitions, size limits, and submission mechanisms accepted by major search engines.

Brass-SEO draws on:

The 50,000 URL limit and lastmod accuracy guidance — cited in Brass-SEO's indexing checks and site crawl analysis.

Primary source

sitemaps.org/protocol.html

Brass-SEO draws on this to interpret robots.txt files accurately. RFC 9309 is the first formal IETF standard for the robots exclusion protocol, published August 2022 after 27 years of de facto practice dating to the 1994 original. The standard codifies that when allow and disallow rules conflict at the same path length, allow takes precedence — a rule that some popular interpretations had inverted. It also sets a formal 512 KiB fetch limit; crawlers conforming to the RFC are not required to parse bytes beyond that threshold.

Examines:

IETF formal standard defining robots.txt syntax, precedence rules, the 512 KiB fetch limit, and crawler compliance expectations.

Brass-SEO draws on:

The allow-precedence rule and 512 KiB limit — cited when Brass-SEO's page audit identifies robots.txt configurations that may unintentionally block crawlers.

Primary source

rfc-editor.org/rfc/rfc9309

Brass-SEO uses this as the specification for push-based indexing notification. IndexNow allows sites to notify participating search engines of new, updated, or deleted URLs in real time, with batches of up to 10,000 URLs per POST request. One submission notifies all participating engines simultaneously because they share submissions with each other. Googlebot is the notable non-participant as of 2025. Brass-SEO implements IndexNow for immediate indexing notification on content publish events.

Examines:

The IndexNow protocol specification — push-based URL change notification, batch limits (10,000 URLs per request), key verification, and the list of participating engines.

Brass-SEO draws on:

The protocol implementation details — Brass-SEO's IndexNow integration is built directly to this specification.

Primary source

indexnow.org/documentation

Brass-SEO cites this when explaining Googlebot's core behaviors. Google's documentation states that Googlebot fetches only the first 2 MB of any file — content or navigation beyond that threshold may not be crawled. Google indexes primarily from the Smartphone Googlebot user agent; the mobile version of a page determines indexing status for both mobile and desktop queries. Verifying Googlebot by reverse DNS lookup — not user-agent string — is the only reliable method, since user-agent strings are trivially spoofed.

Examines:

Google's official technical documentation for Googlebot — crawl behavior, the 2 MB file size limit, mobile-first indexing, and verification methodology.

Brass-SEO draws on:

The 2 MB file limit and mobile-first indexing behavior — cited in Brass-SEO's crawl and page audit recommendations.

Primary source

developers.google.com/search/docs/crawling-indexing/googlebot

Brass-SEO monitors this for crawlability benchmark data. The 2025 Web Almanac SEO chapter benchmarks robots.txt adoption, sitemap usage, and — for the first time at scale — llms.txt and AI crawler directive adoption across millions of analyzed origins from real Chrome user data and the HTTP Archive. Sites that deploy llms.txt and explicit AI crawler directives currently occupy an uncrowded signal space; waiting for mainstream adoption eliminates the early differentiation advantage.

Examines:

Annual independent benchmark of web crawlability signals — robots.txt, sitemap adoption, llms.txt adoption, and AI crawler directives — across millions of origins.

Brass-SEO draws on:

The crawl signal adoption benchmarks — used to contextualize how common or rare a site's crawl configuration is relative to the broader web.

Primary source

almanac.httparchive.org/en/2025/seo

Brass-SEO draws on this to scope crawlability recommendations by site size. Google's crawl budget documentation defines crawl budget as the product of crawl capacity limit (Googlebot's server-load tolerance for a host) and crawl demand (Google's interest in re-crawling URLs based on popularity and freshness signals). For most sites under a few thousand pages with fast server response, crawl budget is not a meaningful constraint — the practical priority is clean canonicals, short redirect chains, and strong internal linking. Crawl budget becomes relevant for sites with hundreds of thousands of URLs or significant duplicate content at scale.

Examines:

Google's official documentation defining crawl budget — the two-factor model (capacity limit × crawl demand) and which site sizes and configurations require active budget management.

Brass-SEO draws on:

The two-factor crawl budget model — informs how Brass-SEO prioritizes crawl-related recommendations based on a site's scale.

Primary source

developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget

Brass-SEO cites this when explaining why a page can rank and still be absent from AI answers. Google's documentation states that the nosnippet directive will also prevent the content from being used as a direct input for AI Overviews and AI Mode, and that max-snippet:[number] will also limit how much of the content may be used as a direct input for those features. max-snippet:0 is equivalent to nosnippet; max-snippet:-1 lets Google choose the length; the data-nosnippet HTML attribute excludes specific passages on span, div, and section elements. A blanket nosnippet or a low max-snippet is the on/off switch for whether Google's AI surfaces can quote a page — it lives in the page's own markup, not robots.txt, and it is easy to leave on by accident.

Examines:

Google's official robots meta tag and X-Robots-Tag documentation — the nosnippet, max-snippet, and data-nosnippet directives and their stated effect on AI Overviews and AI Mode.

Brass-SEO draws on:

The stated nosnippet and max-snippet effect on AI Overviews input — cited when Brass-SEO diagnoses why a ranking page is missing from AI-generated answers.

Primary source

developers.google.com/search/docs/crawling-indexing/robots-meta-tag