The robots txt disallow directive is the most-used and most-misunderstood line in technical SEO. It looks like an off switch. It is not. Disallow controls whether a compliant crawler requests a URL — nothing more. Understanding that single distinction prevents the majority of robots.txt disasters I have cleaned up.
Here is the full behavior: matching rules, precedence, what belongs behind a disallow, and the failure modes that only appear weeks later in Search Console.
The Basic Form
A disallow rule lives inside a user-agent group and takes a path prefix as its value:
User-agent: *
Disallow: /checkout/
That blocks any URL whose path begins with /checkout/. Path matching is prefix-based and case-sensitive, so /Checkout/ stays crawlable. The value must start with a slash; a bare Disallow: checkout is malformed and gets discarded.
Two special values are worth memorizing. Disallow: / blocks the entire host — this is the line that takes a site dark, and it is astonishingly easy to push from staging to production. Disallow: with no value blocks nothing and is the standards-compliant way to say “crawl everything.”
User-Agent Groups and Which One Applies
The robots txt user agent line determines who a group of rules applies to. Crawlers do not merge groups. Each bot finds the single most specific group that names it and obeys only that one, ignoring every other group in the file — including the wildcard.
User-agent: *
Disallow: /internal/
User-agent: Googlebot
Disallow: /drafts/
In that file, Googlebot crawls /internal/ freely. It matched the Googlebot group, so the wildcard group no longer applies to it. If you want Googlebot blocked from both directories, both disallows must be repeated inside the Googlebot group. This is the single most common logic error in real-world robots.txt files.
A practical consequence: adding a named group for one crawler quietly changes the rules for that crawler across your entire file. If you add User-agent: GPTBot with a single disallow, GPTBot now ignores every other restriction you had in place. Audit the whole file whenever you add a group, not just the lines you typed.
Names are matched case-insensitively and without version suffixes, so User-agent: googlebot works fine. Google’s image and news crawlers have their own tokens and follow the same specificity logic.
Wildcards, Anchors, and Query Strings
Google, Bing, and most major crawlers support two pattern characters inside path values. An asterisk matches any run of characters. A dollar sign anchors the match to the end of the URL.
User-agent: *
Disallow: /*?sort=
Disallow: /*.pdf$
Disallow: /tag/*/page/
The first blocks any URL containing a sort parameter. The second blocks PDFs specifically — without the $, it would also block /report.pdf.html. The third blocks paginated tag archives at any depth. Parameter order matters in the raw URL string, so /*?sort= will not match /shoes?color=red&sort=price; use /*sort= if you need it regardless of position.
When Allow Beats Disallow
The Allow directive carves exceptions out of a broader block. When both an Allow and a Disallow match the same URL, Google picks the rule with the longer path. If they are exactly the same length, the less restrictive rule wins.
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
The Allow path is longer, so admin-ajax.php stays crawlable while the rest of the admin directory does not. That specific pair is worth having on any WordPress site, because plenty of front-end functionality routes through that file and blocking it can break rendering.
What to Disallow — and What Not To
When people ask about robots txt what to disallow, the honest answer is: less than you think. Every block is a blind spot. Reasonable candidates share one trait — they generate URLs with no unique value that multiply without limit:
- Faceted filter combinations on category pages
- Internal site-search result URLs
- Session ID and tracking parameter variants
- Cart, checkout, and logged-in account paths
- Infinite calendar or date-range spaces
Note what these have in common: they are crawl-efficiency problems, not visibility problems. If a URL is merely low quality rather than infinite, a canonical tag or a noindex tag handles it better, because both leave Google able to read the page and understand the relationship.
What to leave alone: CSS, JavaScript, fonts, and images used in layout. Google renders pages before evaluating them, and a page rendered without its assets can be judged on a broken layout. The old advice to block /wp-includes/ and /wp-content/plugins/ comes from a pre-rendering era and should be deleted on sight.
Blocking a Whole Site by Accident
The worst robots.txt outage I have seen took eleven days to notice and four months to fully recover. A staging environment had the standard two-line block, someone promoted the environment’s config rather than just the application, and production started serving a bare Disallow: / under a wildcard user agent.
Nothing looks broken when this happens. The site loads normally for humans. Traffic does not crater instantly, because indexed pages stay indexed for a while — rankings erode gradually as Google’s cached understanding of each page goes stale. By the time the graph is obviously wrong, weeks have passed.
Two cheap defenses. Add a deploy-time assertion that fetches production /robots.txt and fails the build if it contains a wildcard group with Disallow: /. And protect staging with HTTP authentication rather than robots.txt, which is both more secure and removes the file that keeps getting promoted by mistake.
The Trap: Disallow Does Not Mean De-Index
Blocking a URL prevents crawling. It does not prevent indexing. If other sites link to a blocked URL, Google can index it from those links alone and show it in results as a bare URL with no title and no snippet. That is the “Indexed, though blocked by robots.txt” status in Search Console.
Worse, disallow actively defeats de-indexing. To remove a page from the index, Google must fetch it and see a <meta name="robots" content="noindex"> tag or an X-Robots-Tag: noindex response header. If robots.txt blocks the fetch, that signal is never read, and the page can sit in the index indefinitely.
The correct sequence is the reverse of what feels natural. Allow the crawl, serve noindex, wait for Google to recrawl and drop the URLs, and only then — if you still want to save crawl budget — add the disallow.
Comments, Review Cadence, and Verification
A robots txt comment starts with # and costs nothing. Annotate every non-obvious rule with what it blocks and when it was added. Robots.txt files accumulate archaeology, and an undocumented disallow tends to survive for years because nobody dares delete it.
Date your rules while you are at it. A note reading # added 2026-03 for the faceted nav rollout, revisit after migration gives the next person enough context to remove it safely. Undated rules never get removed, and a robots.txt that only ever grows will eventually block something important by accident.
Verify behavior after every change with Search Console’s URL Inspection tool on a handful of real URLs — three you expect blocked, three you expect crawlable. Syntax checkers confirm the file parses; only live inspection confirms what Googlebot does with a specific URL.
Then watch coverage over time. A disallow that was correct in March can be wrong in September after a URL structure change. SEO Rocket’s full-site crawls flag blocked and unreachable URLs with per-issue evidence — the actual URLs, titles, and H1 text involved — so an over-broad rule shows up as a concrete list of affected pages instead of a slow, unexplained decline. Editing the file is still your job; knowing it broke something shouldn’t take a quarter.
