You can make robots txt in about ninety seconds with a text editor, and most sites should. The file is plain text, the syntax is roughly a dozen keywords, and the whole thing usually fits in fifteen lines. The hard part is not writing it — it is knowing which rules crawlers honor, which ones they silently discard, and which ones will quietly de-index a section of your site.
This guide covers the syntax that works, the placement rules that trip people up, and the directives that survive in blog posts long after search engines stopped supporting them.
What the File Is and What It Controls
Robots.txt is a crawl-control file. It tells compliant bots which URL paths they may request. That is the entire scope. It does not control indexing, it does not protect private data, and it does not enforce anything — a hostile scraper reads your robots.txt as a map of what you consider sensitive and ignores every rule in it.
Treat it as traffic management for well-behaved crawlers. If a URL must never be publicly visible, put it behind authentication. If a URL should be crawled but kept out of search, robots.txt is the wrong tool entirely, and I will come back to why.
Where the File Must Live
Placement is rigid and non-negotiable. The file must sit at the root of a host and be served at exactly /robots.txt. A file at /blog/robots.txt or /assets/robots.txt does nothing at all.
More importantly, robots.txt is scoped per protocol, per host, and per port. These are four separate files as far as crawlers are concerned:
https://example.com/robots.txthttp://example.com/robots.txthttps://www.example.com/robots.txthttps://shop.example.com/robots.txt
Every subdomain needs its own. This catches teams constantly: a staging subdomain gets indexed because someone assumed the root domain’s rules applied everywhere. They do not. Paths are also case-sensitive — Disallow: /Admin/ will not block /admin/.
The Syntax, Line by Line
Records are grouped by user agent. A group starts with one or more User-agent lines and continues with the rules that apply to them. A blank line separates groups.
User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /*?sessionid=
Allow: /cart/help/
Sitemap: https://example.com/sitemap_index.xml
Four things to know about matching. The asterisk in User-agent: * means “any crawler without a more specific group” — a bot only ever obeys one group, the most specific one that names it. Inside path values, * matches any sequence of characters and $ anchors the end of a URL. An empty Disallow: value means allow everything. And when an Allow and a Disallow both match a URL, Google applies the more specific rule, measured by path length; a tie resolves in favor of allowing.
Comments start with # and run to the end of the line. Use them. Six months from now nobody will remember why /tmp-2024/ was blocked.
What Is Worth Blocking
The instinct is to block a lot. Resist it. Every disallow is a place Google cannot look, and blocking the wrong path does far more damage than leaving a low-value URL crawlable. Sensible candidates:
- Faceted-navigation parameter combinations that generate near-infinite URL variants
- Internal search results pages
- Cart, checkout, and account URLs
- Genuinely infinite spaces — calendars that generate a page for every future date
A useful mental test before adding any rule: if this URL were crawled, what is the worst thing that happens? Usually the answer is “Google spends a few requests on a page nobody searches for,” which costs almost nothing on a site under 10,000 URLs. Crawl budget only becomes a real constraint at scale, and blocking rules aimed at a problem you do not have are pure downside.
What should almost never be blocked: CSS and JavaScript files. Google renders pages, and a page rendered without its stylesheet or scripts can look broken or empty to the renderer. Blanket-blocking /wp-content/ or /assets/ is a legacy habit that actively hurts modern sites.
Directives That No Longer Work
Two lines appear in nearly every generator output and neither does what people think.
Noindex: in robots.txt is not supported. Google stopped honoring it on 1 September 2019, and it was never part of the standard. If you want a page out of the index, allow it to be crawled and serve a <meta name="robots" content="noindex"> tag or an X-Robots-Tag: noindex HTTP header. This is the trap worth memorizing: a URL blocked in robots.txt cannot be fetched, so Google can never see the noindex you put on it. Blocking and de-indexing are opposite actions.
Crawl-delay: is ignored by Google. Bing does honor it. If Googlebot is genuinely overloading your server, adjust crawl rate through Search Console or, better, fix the capacity problem — a server that buckles under a crawler will buckle under a traffic spike too.
The Sitemap Line and Other Supported Extras
Beyond user-agent grouping and path rules, one extension is universally supported and genuinely worth including: Sitemap:. It takes an absolute URL, sits outside any user-agent group, and can be repeated as many times as you need.
Sitemap: https://example.com/sitemap-posts.xml
Sitemap: https://example.com/sitemap-products.xml
This is not a substitute for submitting sitemaps in Search Console, but it does help crawlers that never touch your Search Console property — Bing, and every third-party tool that reads robots.txt as a discovery starting point. Relative paths do not work here; the URL must be absolute, including the protocol.
Keep the file small. Google fetches up to 500 kibibytes of robots.txt and ignores anything past that limit. In practice you will never approach it unless something is generating rules programmatically, which is itself a warning sign — a robots.txt with 2,000 disallow lines is usually a URL architecture problem being papered over.
Generators, Templates, and Why I Skip Them
Search for a robots txt generator free of charge and you will find dozens of online robots txt generator tools that produce the same output: a wildcard user-agent, a handful of CMS-specific disallows, a crawl-delay line, and sometimes a noindex line. Two of those are dead directives and the CMS disallows are often stale by several major versions.
A robots txt file creator is fine as a scaffold if you already know the syntax well enough to delete what it gets wrong. If you do not, you are shipping someone else’s assumptions to your root directory. Writing eight lines by hand takes less time than auditing generated output, and you will understand every line you ship.
Test It, Then Watch It
After you deploy, request the file in a browser and confirm it returns HTTP 200 with text/plain. A robots.txt served as HTML, or returning a soft 404, is a common CDN misconfiguration. Note the edge case: a 5xx response on robots.txt causes Google to pause crawling the host, which is a much worse failure than having no file at all.
Then verify behavior rather than syntax. Search Console’s URL Inspection tool tells you whether a specific live URL is blocked, which is the only question that matters. Spot-check three URLs you expect to be blocked and three you expect to be crawlable.
Ongoing, the useful signal is crawl coverage across the whole site, not the file in isolation. SEO Rocket’s full-site crawls surface blocked and unreachable URLs with concrete per-issue evidence — actual URLs, titles, and H1 text — so a bad disallow shows up as a list of pages rather than a suspicion. It will not edit your robots.txt or server config; that stays your call. But knowing which 340 pages went dark after a deploy beats finding out from a traffic chart six weeks later.
