Your robots.txt file quietly controls which pages search engines are allowed to crawl, and a single misplaced line can block your entire site from Google. Learning to test robots.txt before and after changes is basic technical-SEO hygiene that prevents some of the most damaging, and most avoidable, mistakes. This guide covers how the rules work, how conflicts resolve, and how to check any URL.
What robots.txt does
Robots.txt is a plain text file at the root of your domain (example.com/robots.txt) that tells crawlers which paths they may or may not request. It uses User-agent lines to target specific bots and Allow / Disallow lines to permit or block paths. Important nuance: robots.txt controls crawling, not indexing. A blocked page can still appear in results (without a description) if other sites link to it, to keep a page out of the index, use a noindex meta tag instead.
How Google resolves conflicting rules
When more than one rule matches a URL, Google does not simply read top to bottom. It applies two rules that trip people up constantly:
- The most specific (longest) rule wins. Between
Disallow: /blogandAllow: /blog/public, the longer, more specific path decides the outcome for matching URLs. - On a tie, Allow beats Disallow. If an Allow and a Disallow rule match with equal specificity, the URL is crawlable.
Wildcards matter too: * matches any sequence of characters and $ anchors the end of a URL, so Disallow: /*.pdf$ blocks PDF files specifically.
The mistakes that block your whole site
A handful of errors cause outsized damage:
| Mistake | Effect |
|---|---|
Disallow: / left in from staging |
Blocks the entire site from all crawlers |
| Blocking CSS or JS folders | Google cannot render the page and may misjudge it |
| Using robots.txt to hide a page from search | The page can still be indexed; use noindex instead |
| Case mismatch in a path | Rules are case-sensitive, /Blog is not /blog |
| No Sitemap directive | A missed, easy signal, add Sitemap: https://.../sitemap.xml |
How to test a specific URL
Never guess whether a URL is blocked, test it. Our free robots.txt tester lets you paste your robots.txt, enter a URL and a crawler name, and it applies Google’s real longest-match rules to tell you whether that URL is allowed or blocked, and which rule decided it. That makes it easy to confirm a change does what you intended before it goes live. If you are creating the file from scratch, the robots.txt generator builds a valid one.
Test after every change
Robots.txt is high-stakes precisely because it is so small: there is no visual feedback when you get it wrong, just a slow, silent drop in crawling and rankings. Make it a habit to test the file whenever you edit it, and to spot-check a few important URLs (your homepage, a key category, a money page) to confirm they are crawlable. Also check that you have not accidentally blocked resources the page needs to render.
robots.txt versus meta robots versus X-Robots-Tag
Robots.txt is only one of three ways to control crawlers, and using the wrong one is the root of most “why is this page still showing up?” problems. Here is when to use each:
| Method | Controls | Use it to |
|---|---|---|
| robots.txt | Crawling | Stop bots requesting whole sections (e.g. admin, faceted URLs) |
| meta robots (noindex) | Indexing | Keep a specific HTML page out of the search index |
| X-Robots-Tag header | Indexing | Apply noindex to non-HTML files like PDFs and images |
The critical interaction: if you Disallow a page in robots.txt, Google cannot crawl it, which means it cannot see a noindex tag on that page, so the page can stay indexed. To reliably remove a page from search, allow crawling and add noindex; do not block it in robots.txt. Match the tool to the goal: robots.txt to save crawl budget, noindex to control the index.
Do not forget AI crawlers
Robots.txt is now also where you decide whether AI crawlers like GPTBot, ClaudeBot and PerplexityBot may access your content. Whether you allow or block them is a strategic choice, but you should make it deliberately rather than by accident. Our AI crawler robots.txt checker shows which AI bots your current file allows or blocks.
The takeaway
Robots.txt controls crawling, resolves conflicts by longest-match with Allow winning ties, and can block your whole site with one stray line. Test every change against real URLs before trusting it. Check yours now with the robots.txt tester, and explore the rest of our free SEO tools for the rest of your technical setup.