Robots.txt: The File That Can Hide Your Site from Google

Q: My CMS manages robots.txt automatically. Should I worry about it?

Probably not, but it is worth checking once. Go to yourdomain.com/robots.txt and verify it looks reasonable. Most CMS platforms (WordPress, Squarespace, Shopify) generate sensible defaults. The risk comes from plugins, custom configurations, or settings that were changed and forgotten.

There is a plain text file on your website called robots.txt. It sits at the root of your domain — yourdomain.com/robots.txt — and it tells search engine crawlers what they are and are not allowed to access.

Most business owners have never looked at this file. Most do not know it exists. But a single misconfigured line can block Google from seeing your entire website, individual pages, or critical resources like CSS and JavaScript that Google needs to render your pages correctly.

Here is what robots.txt does, how to check yours, and the mistakes that can make you invisible.

What Robots.txt Does
How to Check Your Robots.txt
What a Normal Robots.txt Looks Like
Dangerous Mistakes
When to Edit Your Robots.txt
Frequently Asked Questions

What Robots.txt Does

Robots.txt is a standard protocol (formally called the Robots Exclusion Protocol) that tells web crawlers — like Googlebot — which URLs they are allowed to request from your site.

What it controls:

Which pages Googlebot can crawl
Which directories are off-limits
Where your sitemap is located
Different rules for different crawlers

What it does NOT control:

Whether a page gets indexed (a page can be indexed without being crawled if other sites link to it)
Your search rankings (robots.txt is not a ranking signal)
What humans can access (it only affects well-behaved bots)

Think of it as a set of suggestions for search engine crawlers. Google respects robots.txt directives, but they are not security measures — anyone can still visit a URL directly.

How to Check Your Robots.txt

Type yourdomain.com/robots.txt into your browser. You will see a plain text file. If you see a 404 error, you do not have a robots.txt file — which is fine. No robots.txt means "crawl everything," which is the default most small business sites want.

You can also check in Google Search Console. Go to Settings > Crawl Stats > robots.txt. This shows what Google sees and when it last fetched your robots.txt.

What a Normal Robots.txt Looks Like

Here is a typical robots.txt for a small business site:

User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

This says: "All crawlers are allowed to access everything. Here is where the sitemap is."

A slightly more restrictive version:

User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/

Sitemap: https://yourdomain.com/sitemap.xml

This blocks crawlers from admin pages, cart pages, checkout flows, and account pages — all pages that should not appear in search results.

Dangerous Mistakes

Mistake 1: Blocking Your Entire Site

User-agent: *
Disallow: /

These two lines tell every crawler to stay away from every page. Your entire site disappears from search results. This sometimes happens when a developer adds this during development (to prevent Google from indexing an unfinished site) and forgets to remove it before launch.

How to spot it: If your site suddenly has zero search traffic and no pages in Google's index, check robots.txt first.

Mistake 2: Blocking Important Pages

User-agent: *
Disallow: /services/

This blocks all pages in the /services/ directory. If that is where your service pages live, none of them will appear in search results. This can happen accidentally when someone means to block one specific page but uses a directory-wide rule.

Mistake 3: Blocking CSS and JavaScript

User-agent: *
Disallow: /wp-content/themes/
Disallow: /wp-includes/

This blocks Google from accessing your CSS and JavaScript files. Google needs these to render your page correctly. Without them, Google sees raw HTML that may look completely different from what users see. This can cause Google to misinterpret your page layout and content.

Mistake 4: Using Robots.txt for Security

Robots.txt is public. Anyone can read it by visiting yourdomain.com/robots.txt. Listing sensitive URLs in your Disallow directives actually reveals those URLs to anyone looking. If you have pages that truly need to be hidden, use proper authentication or server-side access controls — not robots.txt.

When to Edit Your Robots.txt

Most small business sites never need to touch robots.txt, but a few situations call for a careful edit.

Edit it if:

You have admin or backend pages that should not be crawled
A directory contains pages with duplicate or thin content (e.g., search result pages on your site)
You want to prevent crawling of specific file types (PDFs, images) from certain directories
You need to add your sitemap URL

Do NOT edit it if:

You want to remove a page from search results (use a noindex meta tag instead)
You are trying to hide content for security reasons
You are not sure what a change will do (test first)

How to test changes: Use GSC's robots.txt Tester (Settings > robots.txt) to verify that your changes allow and block the right URLs before making them live.

For checking whether any of your important pages are blocked from Google's crawl, the indexing check in Brass-SEO can identify pages with indexing issues, including those blocked by robots.txt. Combined with your GSC data, you can see exactly which pages Google can and cannot access.

Frequently Asked Questions

What happens if I do not have a robots.txt file?

Nothing bad. No robots.txt file means Google crawls everything on your site, which is the default behavior most small business sites want. A missing robots.txt is not an error. You only need one if you want to restrict crawling of specific pages or directories.

Can I use robots.txt to deindex a page?

No. Robots.txt blocks crawling, not indexing. If other sites link to a page that is blocked by robots.txt, Google may still index the URL (without page content) based on the anchor text of those external links. To remove a page from search results, use a noindex meta tag on the page itself. Robots.txt and noindex serve different purposes.

How quickly does Google respond to robots.txt changes?

Google fetches your robots.txt periodically — usually every few hours to once a day for active sites. Changes take effect the next time Google fetches the file. If you make an urgent change, you can request a re-crawl of specific pages through GSC's URL Inspection tool.

My CMS manages robots.txt automatically. Should I worry about it?

Probably not, but it is worth checking once. Go to yourdomain.com/robots.txt and verify it looks reasonable. Most CMS platforms (WordPress, Squarespace, Shopify) generate sensible defaults. The risk comes from plugins, custom configurations, or settings that were changed and forgotten.

Is it possible for robots.txt to improve my SEO?

Indirectly. By blocking low-value pages from crawling, you help Google spend its crawl budget on your important pages. This matters more for large sites (thousands of pages) than for small business sites. For a site with 20-50 pages, crawl budget optimization via robots.txt is rarely necessary.