Google retired the legacy robots txt tester inside Search Console in December 2023, replacing it with a slimmer robots.txt report. If you learned this workflow before then, the tool you are looking for no longer exists in the form you remember — and a lot of the third-party alternatives that call themselves a robots txt validator are only checking syntax, not behavior.
Those are different questions. Syntax validation asks whether the file parses. Behavior testing asks whether a specific URL is crawlable. Only the second one predicts what happens in search.
What Replaced the Old Tester
Search Console now offers a robots.txt report under Settings. It shows the files Google fetched for your property, when each was last fetched, the HTTP status returned, and any parse errors or warnings Google found. It is a monitoring surface, not an interactive sandbox — you cannot paste in a hypothetical rule and see what it would do.
The report is genuinely useful for the failure it catches best: a robots.txt returning the wrong status code. A 5xx response on robots.txt causes Google to pause crawling the entire host, which is a far more expensive outage than a wrong Disallow line. Watching fetch status here catches that within a day.
URL Inspection Is the Real Tester
To verify robots txt behavior for a live URL, use URL Inspection. Paste any URL on your verified property and Google will tell you, unambiguously, whether crawling is allowed and what it saw on the last fetch. This is the closest thing to ground truth available, because it reflects Googlebot’s own evaluation rather than a third-party reimplementation of the matching rules.
Make a habit of checking six URLs after any robots.txt change — three you expect blocked, three you expect crawlable. It takes two minutes and it catches wildcard rules that matched more than intended, which is the most common and most costly error.
Precedence is where hand-reasoning goes wrong most often, so test it rather than deduce it. When an Allow and a Disallow both match a URL, Google does not take the first rule in the file — it takes the most specific one, measured by the length of the path pattern, and breaks ties in favor of allowing. Order in the file is irrelevant. Wildcards count toward that length calculation too, which makes patterns like Disallow: /*.php$ versus Allow: /admin/ genuinely hard to predict on paper.
The limitation is that inspection works only on verified properties and only on live URLs. You cannot test a rule before deploying it, which brings us to local testing.
Testing Rules Before You Deploy
Google open-sourced its production robots.txt parser in 2019. That C++ library is the same matching logic Googlebot uses, and it is available to run locally. There are also maintained Python and Node ports that follow the same specification closely. Feeding a candidate file and a list of URLs through one of these gives you a genuine pre-deploy test rather than an approximation.
Build a small regression suite while you are there: a list of twenty URLs paired with the expected verdict, blocked or allowed. Run the candidate file against it before every change. It takes an hour to write and it will catch the wildcard that matched three extra directories long before Googlebot does.
If you would rather not compile anything, the practical alternative is staging. Deploy the new robots.txt to a staging host, crawl it with any desktop crawler configured to respect robots.txt, and compare the crawlable URL count against production. A large unexpected drop is your answer.
Where Third-Party Checkers Help and Where They Mislead
Search for a robots txt checker online and you will find dozens of tools. The good ones do three things well: fetch the file and confirm it returns 200 with a text/plain content type, flag malformed lines, and let you test a URL against the ruleset. That covers most day-to-day needs.
Where they mislead is subtler. Several popular tools still flag missing Crawl-delay or missing Noindex lines as issues to fix. Both directives are unsupported by Google — crawl-delay is ignored outright, and Noindex: in robots.txt stopped working on 1 September 2019. A checker recommending either is working from a stale rulebook and will lead you to ship dead lines.
Others implement wildcard matching loosely, which means a URL that a tool reports as blocked may in fact be crawlable, or vice versa. When a third-party result and URL Inspection disagree, believe URL Inspection.
A Practical Test Checklist
Whatever tool you use, these are the checks that actually matter. Run them after every change to the file:
- The file is served at exactly
https://yourdomain.com/robots.txt, returns HTTP 200, and has atext/plaincontent type - Every subdomain and protocol variant you use has its own file — robots.txt is per-host, per-protocol, per-port, and rules do not inherit
- No bare
Disallow: /under a wildcard user agent, unless you mean it - CSS, JavaScript, and layout images are crawlable — Google renders pages and blocked assets can make a page look broken
- Paths match your actual URL casing; robots.txt paths are case-sensitive
- Each named user-agent group repeats every rule it needs — a bot obeys only its own group and ignores the wildcard entirely
- No
Noindex:orNofollow:lines; move those to a meta robots tag or anX-Robots-Tagheader - The
Sitemap:line uses an absolute URL
How to Find and Read Someone Else’s
To find robots txt for any site, append /robots.txt to the root domain. That is it — the file is public by design, and there is no other valid location. Reading a competitor’s file is a legitimate and quick piece of research: it often reveals internal directory structure, parameter conventions, staging hosts left in comments, and which crawlers they are actively fighting.
It also shows you their sitemap URLs, which is the fastest way to size their published footprint. Do not copy their rules, though. Their disallows are shaped by their CMS and their URL patterns, not yours.
Common Failures a Test Will Catch
Across audits, the same handful of problems account for most robots.txt damage. Knowing them turns testing from a vague sweep into a targeted check.
- The wildcard group is ignored. A file with a named
User-agent: Googlebotgroup means Googlebot obeys only that group. Rules you assumed applied from the wildcard group do not. - A trailing rule matched more than intended.
Disallow: /problocks/products/,/promotions/, and/profile/. Prefix matching has no implicit word boundary. - The CMS is generating a virtual file. WordPress serves a dynamic robots.txt when no physical file exists, and several plugins overwrite it. What you edited on disk may not be what is served.
- The CDN is serving a cached copy. Changes look live at origin and stale at the edge for hours. Always test the public URL, not the origin.
- A redirect sits in front of the file. Google follows up to five redirects for robots.txt; beyond that it treats the file as unavailable.
Each of these produces a file that validates cleanly. That is precisely why syntax checking alone gives false confidence.
Testing the File Versus Testing the Outcome
A valid robots.txt can still be the wrong robots.txt. The file might parse perfectly while quietly blocking 3,000 product pages because a wildcard matched a path you forgot about. Syntax testing will never catch that; only crawling the site will.
That is the check worth automating on a schedule. SEO Rocket runs full-site crawls — quick scans of roughly 25 pages, or deep crawls verified past 900 — and reports blocked and unreachable URLs with concrete evidence per issue, showing the actual URLs, titles, and H1 text involved. It does not edit your robots.txt or server config, and it shouldn’t; those are deliberate changes with real consequences. What it does is turn “the file looks fine” into a list of pages that either are or are not reachable.
