Most people treat robots txt as a privacy setting — block a URL here and it disappears from Google. That belief has quietly deindexed important pages and, just as often, left “hidden” URLs sitting in the search results anyway. A robots.txt file does exactly one thing: it tells compliant crawlers which paths they may or may not fetch. It says nothing about indexing, ranking, or secrecy. Once you internalize that crawling and indexing are separate systems, most robots txt mistakes stop happening — and the file becomes a precision tool for steering crawl budget instead of a blunt object that breaks things.
What a Robots.txt File Actually Is
Robots.txt is a plain UTF-8 text file that lives at the root of a host — https://example.com/robots.txt — and follows the Robots Exclusion Protocol, formalized as RFC 9309 in 2022. When a crawler wants to visit your site, it fetches that file first and reads the rules for its own user-agent. There’s no registration, no API; the file’s location is the contract. It’s advisory, not enforced: Googlebot, Bingbot, and other major crawlers obey it, but a hostile scraper can ignore it entirely. That’s the first thing to get straight — robots.txt is a traffic-routing convention for well-behaved bots, never a security control.
Crawling vs Indexing: The Distinction Everything Hinges On
Search visibility runs through three separate stages: crawling (a bot fetches the URL), rendering (it executes the page’s JavaScript to see the final DOM), and indexing (it decides whether to store the page for retrieval). Robots.txt only touches the first stage. A Disallow rule stops Googlebot from fetching a URL — but Google can still index that URL from external signals alone. If ten sites link to /secret-page/ with descriptive anchor text, Google may list it in results as a URL-only entry with no snippet, showing “No information is available for this page.” You blocked the crawl and got the opposite of what you wanted: a visible, uncontrollable listing.
The Noindex Trap Nobody Warns You About
This is the single most expensive robots.txt error, so it gets its own section. To keep a page out of the index, you use a noindex directive — either a <meta name="robots" content="noindex"> tag or an X-Robots-Tag: noindex HTTP header. But Googlebot can only obey a noindex it can actually see, and it can only see it by crawling the page. If you Disallow that same URL in robots.txt, you’ve blocked the very crawl that would deliver the noindex instruction. The two directives fight each other and disallow wins the crawl, so the noindex is never read and the page can linger in the index indefinitely.
The rule that fixes this is simple: never disallow a URL you’re trying to noindex. Let Google crawl it, read the noindex, and drop it. Only after it has fully deindexed can you optionally block it to save crawl budget. Note too that Google formally dropped support for an unofficial noindex: line inside robots.txt back in September 2019 — if you have one, it does nothing.
Anatomy of the Directives
A robots.txt file is a set of groups, each opening with one or more User-agent lines followed by rules. The directives that actually matter:
- User-agent — names the crawler the group applies to.
*is the catch-all for any bot without a more specific group. - Disallow — a path prefix the bot must not crawl. An empty
Disallow:means “nothing blocked.” - Allow — carves an exception out of a broader Disallow. Google and Bing support it; it’s essential for patterns like “block the folder but permit one file inside it.”
- Sitemap — an absolute URL to your XML sitemap. It’s independent of user-agent groups and can sit anywhere in the file.
Two things Google has retired: crawl-delay is ignored by Googlebot (control crawl rate in Search Console or by server response instead), and there’s no valid nofollow directive here. Keep the file to the directives that are actually honored.
A Real, Correct Example
Here’s a robots.txt file that blocks the noise most sites should block while keeping the assets Google needs to render pages:
User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /*?sort=
Disallow: /search
Allow: /search/help
Sitemap: https://example.com/sitemap.xmlNotice what’s not blocked: CSS and JavaScript directories. Blocking those is a classic own-goal — Google renders pages before judging them, and if it can’t fetch your stylesheets and scripts, it may see a broken layout and misjudge mobile-friendliness or content. Let render-critical resources through.
Wildcards, Anchors, and Matching Rules
Google’s parser supports two special characters that make pattern-blocking precise. The asterisk * matches any sequence of characters, and the dollar sign $ anchors a rule to the end of the URL. So Disallow: /*.pdf$ blocks every URL ending in .pdf, while Disallow: /*? catches any URL containing a query string. When multiple rules match the same URL, Google applies the most specific rule — the one with the longest matching path. If an Allow and a Disallow are exactly the same length, the Allow wins. Paths are also case-sensitive: /Page and /page are different URLs, so match the real casing your CMS produces.
Robots.txt and Crawl Budget
For large sites, this is where the file earns its keep. Crawl budget is the finite number of URLs Googlebot will fetch from your site in a given window. Every wasted fetch on a faceted-navigation permutation, an internal search results page, or a session-ID URL is a fetch not spent on a page that matters. Using Disallow to prune those low-value URL families — infinite filter combinations, calendar pages stretching to the year 3000, printer-friendly duplicates — concentrates crawling on your real content and speeds up discovery of new and updated pages.
The honest caveat: crawl budget is only a live concern for sites with hundreds of thousands of URLs or a sprawling parameter space. A 200-page brochure site does not have a crawl-budget problem, and blocking things to “save budget” there just adds risk. Diagnose before you prune — server log files and Search Console’s Crawl Stats report show you where Googlebot actually spends its time, so you block the real waste rather than guessing.
The Mistakes That Actually Deindex Sites
A handful of robots txt errors do outsized damage, and they cluster around deployment:
- The staging block shipped to production. A
Disallow: /is standard on staging environments, and pushing that file live is the fastest way to erase a site from Google. Diff your robots.txt on every deploy. - Serving a 5xx on the robots.txt request. If Google repeatedly can’t fetch the file (server error), it may treat the whole site as disallowed and slow or stop crawling. A 404 is safer — Google reads a missing file as “crawl everything.”
- Blocking CSS/JS and starving the render, as covered above.
- Disallowing a URL that should be noindexed — the trap that keeps unwanted pages in the index.
These are exactly the class of issue that hides until traffic craters, because nothing on the page looks broken. This is where continuous auditing pays off. Rather than remembering to eyeball the file, SEO Rocket’s real-crawler site audit fetches your pages the way a bot does and flags robots.txt conflicts — blocked resources, disallow-plus-noindex contradictions, an unreachable file, orphaned directives — with the fix spelled out, so a bad deploy surfaces in hours instead of after a ranking cliff.
Subdomains, Protocols, and Scope
Robots.txt is scoped to a single origin — the exact protocol, host, and port. https://example.com/robots.txt governs only that host. Your blog on blog.example.com needs its own file at https://blog.example.com/robots.txt; the root domain’s rules don’t cascade to subdomains. Likewise http:// and https:// are technically distinct origins. And the file must sit at the root path — a robots.txt in a subfolder is simply ignored. Get the placement wrong and your carefully written rules apply to nothing.
Testing and Monitoring Your Robots.txt
Google retired the standalone robots.txt Tester, but the robots.txt report in Search Console (under Settings) shows the version Google last fetched, when it fetched it, and any parse errors — and lets you request a re-crawl after edits. Because Googlebot caches robots.txt for roughly 24 hours, a change you make now won’t take effect instantly; the report tells you which version is currently authoritative. For a quick manual check, the URL Inspection tool reports whether a specific URL is blocked. On any large or frequently deployed site, treat the file as monitored infrastructure, not a set-and-forget config — a single stray character can gate your whole domain.
Where Robots.txt Fits in a Real SEO Workflow
A good robots txt strategy is one lever among many, and it only matters once the pages behind it are worth crawling. The sequence that scales: research keywords against real search data, build pages that beat the weakest result on page one, keep the architecture crawlable, and then use robots.txt to point crawl budget at the pages that earn traffic. SEO Rocket runs that full loop — AI keyword research on live Ahrefs data, a validation-gated article writer, competitor gap analysis, rank and AI-visibility tracking, and the crawler-based audit that catches robots.txt regressions — for roughly $50/month with a free tier. It’s the playbook proven across 1,000,000+ ranking pages: get the crawl signals right so the good content actually gets found. For the deepest log-file forensics on an enterprise crawl, a dedicated desktop crawler still has its place; SEO Rocket is the continuous, no-setup layer that watches for the everyday robots.txt mistakes that quietly cost rankings.
Frequently Asked Questions
Does robots.txt stop a page from appearing in Google?
No. A Disallow rule stops Google from crawling the page, but the URL can still be indexed from external links and shown as a bare listing with no description. To keep a page out of results, allow the crawl and use a noindex meta tag or X-Robots-Tag header instead — never disallow a URL you want deindexed.
Where does the robots txt file have to be located?
At the root of each host: https://yourdomain.com/robots.txt. It only governs that exact protocol and hostname, so subdomains need their own file. A robots.txt placed in any subfolder is ignored completely.
Should I block CSS and JavaScript in robots.txt?
No. Google renders pages before evaluating them, and blocking render-critical CSS or JS can make your page look broken to the crawler, hurting mobile usability and content assessment. Leave those resources crawlable.
Is robots.txt case-sensitive?
The directives (User-agent, Disallow) are not, but the paths are. /Downloads/ and /downloads/ are different URLs, so your Disallow rules must match the exact casing your site serves.
