Most people who set out to make robots txt rules are trying to solve the wrong problem. They want to hide a page from Google, so they block it in robots.txt — and then are baffled when the URL keeps showing up in search results with a “no information is available” snippet. Robots.txt was never an indexing control. It is a crawl-traffic manager, and confusing those two jobs is how a single misplaced line quietly costs a site months of rankings. Get the mental model right first, and the syntax becomes trivial.
What Robots.txt Actually Controls (And the One Thing It Never Will)
Robots.txt tells cooperating crawlers which URL paths they may request. That is the entire job. It manages crawl load — keeping bots out of infinite calendar pages, faceted-search combinations, and internal search results that would otherwise burn crawl budget for zero ranking value.
What it does not do: it does not remove a page from Google’s index, it does not password-protect anything, and it does not stop a determined scraper. If a blocked URL has inbound links, Google can still index the URL itself — it just can’t read the content behind it. So the file that was supposed to hide the page instead publishes a blind, snippet-less listing of it. To actually keep something out of the index, you use a noindex meta tag or X-Robots-Tag header on a page that stays crawlable. Block it in robots.txt and Google never sees the noindex, because it never fetches the page. That single interaction is the most expensive mistake in the whole topic.
Where the File Must Live — This Part Is Non-Negotiable
The file must sit at the root of the host and be named exactly robots.txt, served at https://example.com/robots.txt. A robots file in a subfolder is invisible; crawlers only ever look at the root. It is also specific to protocol, host, and port. These are four separate scopes that each need their own file:
https://example.com/robots.txthttp://example.com/robots.txthttps://www.example.com/robots.txthttps://shop.example.com/robots.txt
Two practical limits matter. Google reads up to 500 KB of a robots.txt file and ignores anything beyond that, so keep it lean. And Google caches the file for roughly 24 hours — a fix you push today may not take effect on the crawler’s side until tomorrow. Plan changes with that lag in mind rather than assuming instant effect.
The Syntax, Decoded Line by Line
A robots.txt file is a set of groups. Each group starts with one or more User-agent lines naming the bots it applies to, followed by Allow and Disallow rules. A minimal, correct file looks like this:
User-agent: *— the rules below apply to every crawler that doesn’t have its own named groupDisallow: /cart/— blocks every path starting with/cart/Disallow: /*?sort=— the*wildcard matches any characters, so this blocks any URL containing?sort=Allow: /cart/policy— carves an exception back out of a broader disallowSitemap: https://example.com/sitemap.xml— an absolute URL, placed anywhere in the file
Two special characters carry most of the weight. The asterisk * matches any sequence of characters; the dollar sign $ anchors a match to the end of the URL, so Disallow: /*.pdf$ blocks PDF files but not a path like /pdf-guide/. An empty Disallow: means “block nothing,” and Disallow: / means “block the entire site” — one stray slash is the difference between open and sealed.
The Matching Rule That Trips Almost Everyone
When an Allow and a Disallow both match the same URL, people assume Disallow wins or that order decides it. Neither is true for Google. The rule is most-specific-wins, measured by the number of characters in the matched path. The longer, more precise rule takes precedence; only when two rules are exactly the same length does the least restrictive one (Allow) win.
Here is a worked micro-example. Suppose you have:
Disallow: /blog/Allow: /blog/public/
A request for /blog/public/post-1 matches both rules. Disallow matches on 6 characters (/blog/); Allow matches on 13 (/blog/public/). Allow is longer, so the page is crawlable — exactly what you wanted. Flip your intuition here and you will either leak crawl budget or, worse, block a section you meant to keep open. When you make robots txt rules with overlapping paths, trace each important URL through this length test before you ship.
What Is Actually Worth Blocking
Blocking is not free — every disallowed path is a section Google can’t evaluate. Reserve it for URLs that generate crawl waste and hold no ranking value:
- Internal search-result pages (
/search?q=) — near-infinite, thin, and duplicative - Faceted-navigation parameter explosions (
?color=&size=&sort=) that spawn thousands of near-identical URLs - Cart, checkout, and account pages that should never rank
- Staging or admin paths (though these need real authentication, not just a Disallow line)
What you must never block: CSS and JavaScript files. Google renders pages like a browser, and if you disallow /assets/ or /wp-includes/, the crawler sees a broken, unstyled page and may misjudge its quality or mobile-friendliness. This is one of the most common self-inflicted wounds a site audit turns up. SEO Rocket’s site audit runs a real crawler against your pages and flags exactly this: important URLs sitting behind a Disallow, or render-critical resources accidentally blocked — the kind of silent error you don’t notice until traffic slides.
Blocking AI Crawlers: The 2026 Wrinkle
A newer reason people want to make robots txt changes is to control AI training and answer-engine bots. These respect robots.txt as a convention, and each has its own user-agent token — GPTBot and OAI-SearchBot (OpenAI), Google-Extended (Gemini/Vertex training, separate from search), ClaudeBot (Anthropic), and CCBot (Common Crawl, which feeds many models). You block them with named groups, for example User-agent: GPTBot followed by Disallow: /.
Two honest caveats. First, robots.txt is voluntary; well-behaved bots obey it, but nothing enforces it, so treat this as a request, not a wall. Second, blocking Google-Extended does not affect your normal Google Search ranking — that is a different crawler — but blocking a search-and-answer bot like OAI-SearchBot can remove you from AI answer surfaces where visibility is increasingly valuable. Decide deliberately whether you’re protecting content or cutting off a growing traffic channel. Tracking how often you actually appear in those AI answers — something SEO Rocket monitors alongside classic rank tracking — turns that from a guess into a measured trade-off.
Directives That Silently Do Nothing
Plenty of robots.txt files carry lines that were never or are no longer supported, and they fail silently — no error, just no effect:
Noindex:— Google stopped honoring this unofficial directive in September 2019. It does nothing. Use a meta robots tag or header instead.Crawl-delay:— Google ignores it entirely (adjust crawl rate in Search Console if needed), though Bing and Yandex do honor it.Host:— a legacy Yandex directive irrelevant to Google.
If you inherited a file stuffed with these, they’re dead weight at best and a false sense of security at worst — someone thinks a page is noindexed when it is merely blocked from crawling, which as we’ve seen is the opposite of what they wanted.
The Sitemap Line and How It Pairs With Meta Robots
Add one or more Sitemap: lines with absolute URLs so crawlers discover your XML sitemap without you submitting it manually. It’s a helpful nudge, not a replacement for submitting the sitemap in Search Console. More importantly, understand which tool does which job, because they’re constantly confused:
- Want a page crawled but not indexed? Keep it out of robots.txt; add
noindexvia meta tag orX-Robots-Tag. - Want to save crawl budget on worthless URLs? Disallow them in robots.txt.
- Want something genuinely private? Neither — put it behind authentication.
Robots.txt and the noindex tag are not interchangeable, and using one where you need the other is the root cause of most “why is this page still showing up” tickets.
Test It, Then Keep Watching It
Before you trust a new file, confirm the basics: it returns an HTTP 200, it’s valid UTF-8 text, and — critically — that a 5xx server error on the robots.txt URL doesn’t silently tell Google to stop crawling your whole site (a prolonged 500 on robots.txt is treated as “disallow everything”). Then spot-check real URLs with Search Console’s URL Inspection tool, which shows exactly whether a given page is blocked and by which rule.
Testing once isn’t enough. CMS updates, plugin changes, and migrations rewrite robots.txt without warning — a staging file with Disallow: / pushed to production has quietly deindexed more sites than any algorithm update. Monitor it as an ongoing signal, not a one-time setup. A crawl-based audit that re-checks the file on a schedule catches the regression in days instead of after a traffic collapse — the playbook we built into SEO Rocket after seeing the same failure across a portfolio proven on 1,000,000+ ranking pages.
Frequently Asked Questions
Does robots.txt stop a page from being indexed?
No. It stops crawling, not indexing. A blocked URL with inbound links can still appear in results without a description. To keep a page out of the index, leave it crawlable and add a noindex meta tag or X-Robots-Tag header.
Do I even need a robots.txt file?
If you have nothing to block, no — a missing file simply means “crawl everything,” which is fine for many small sites. But it’s worth having one to point to your sitemap and to control crawl waste as the site grows. An empty or open file is safer than a wrong one.
Should I use a robots.txt generator?
Be cautious. Many generators inject dead directives (Noindex:, Crawl-delay: for Google) and CMS-specific rules you don’t need. The syntax is simple enough to write by hand, and a hand-written file is easier to audit than boilerplate you don’t understand.
How long until Google respects my changes?
Google caches robots.txt for about 24 hours, so a change can take up to a day to take effect on the crawler’s side. You can request a re-fetch in Search Console, but plan around the lag rather than expecting instant results.
The Bottom Line
To make robots txt work for you rather than against you, hold one distinction in your head: it controls crawling, never indexing. Put the file at the host root, block only URLs that waste crawl budget, never touch your CSS and JavaScript, trace overlapping Allow/Disallow rules through the character-length test, and reach for a noindex tag whenever indexing — not crawling — is the real goal. Then test it, and keep testing it, because the most damaging robots.txt errors are the ones no one notices until the traffic is already gone.
