A robots.txt file is the first thing many crawlers request when they visit your site, and it controls which parts they are allowed to crawl. Learning to create robots.txt correctly is basic technical-SEO hygiene: done right it saves crawl budget and keeps bots out of areas that do not belong in search; done wrong it can hide your entire site from Google. This guide covers the syntax, what to include, and the mistakes to avoid.
What robots.txt is
Robots.txt is a plain text file that lives at the root of your domain, example.com/robots.txt, and gives instructions to web crawlers. It uses a simple format of user-agent groups and allow/disallow rules to say which paths a given bot may or may not request. One crucial point up front: robots.txt controls crawling, not indexing. It stops well-behaved bots from fetching a page, but a blocked page can still appear in search results if other pages link to it. To keep a page out of the index, use a noindex tag instead.
The basic syntax
A robots.txt file is built from a few directives:
| Directive | Meaning |
|---|---|
| User-agent | Which crawler the following rules apply to (* means all) |
| Disallow | A path the bot should not crawl |
| Allow | A path the bot may crawl (often an exception inside a disallowed folder) |
| Sitemap | The full URL of your XML sitemap |
The simplest valid file, User-agent: * followed by an empty Disallow:, means “all bots may crawl everything,” which is the right default for most sites.
What to allow and disallow
Most sites should let crawlers access their content and only block genuinely non-public or low-value areas. Reasonable things to disallow include admin and login paths, internal search-results pages, cart and checkout URLs, and faceted-navigation parameters that generate endless near-duplicate URLs. What you should not block is anything you want indexed, or the CSS and JavaScript files Google needs to render your pages, blocking those can make Google misjudge your layout and mobile-friendliness.
Always add your sitemap
One easy, high-value line many sites forget: add a Sitemap: directive pointing to your XML sitemap’s full URL. This helps every crawler discover your sitemap even if you have not submitted it everywhere. It is a one-line addition that costs nothing and aids discovery, so there is no reason to leave it out. If you have several sitemaps, point to your sitemap index. Not sure your sitemap is valid? Check it with our sitemap validator.
The mistakes that block your whole site
A handful of errors cause outsized damage:
Disallow: /left in from a staging site blocks every crawler from the entire domain, the single most common catastrophic mistake.- Blocking CSS/JS folders stops Google rendering pages properly.
- Using robots.txt to hide a page from search, it does not work; the page can still be indexed. Use noindex.
- Case and path errors, rules are case-sensitive and match from the start of the path.
Create and test your robots.txt
Because a single wrong line has big consequences, generating and testing beats writing from memory. Our free robots.txt generator builds a valid file with your allow, disallow, crawl-delay and sitemap directives, ready to upload to your site root. After you deploy it, confirm it behaves as intended: our robots.txt tester checks whether any specific URL is allowed or blocked for a given crawler, using Google’s real matching rules, so you can be sure you have not accidentally blocked something important.
Do not forget AI crawlers
Robots.txt is now also where you decide whether AI crawlers, GPTBot, ClaudeBot, PerplexityBot and others, may access your content. Whether you allow or block them is a strategic choice, but make it deliberately rather than by accident. Our AI crawler robots.txt checker shows which AI bots your current file allows or blocks so you can decide with full information.
The takeaway
Robots.txt lives at your domain root and tells crawlers which paths they may fetch; it controls crawling, not indexing, so use noindex to keep pages out of search. Block only non-public or low-value areas, never your content or CSS/JS, always include your sitemap, and watch for the stray Disallow: / that hides everything. Build yours with the robots.txt generator, test it with the robots.txt tester, and explore the rest of our free SEO tools.