Guides · July 23, 2026 · 6 min read

Robots.txt Noindex: The Directive Google Retired and What to Use Instead

robots txt noindex

If you are still writing robots txt noindex lines, they are doing nothing. Google announced on 2 July 2019 that it would stop supporting unpublished and unsupported rules in robots.txt, and turned them off on 1 September 2019. The Noindex: directive was never part of the Robots Exclusion Protocol — Google honored it informally for years, then deprecated it as part of standardizing the spec.

The line still shows up in generator output and in advice articles that have not been touched since 2018. Here is what actually removes a page from search, and why the robots.txt approach was doomed by its own logic even when it worked.

What Was Removed and When

Alongside Noindex:, Google dropped support for Nofollow: and Crawl-delay: as robots.txt directives in the same announcement. So a no index robots txt line, a robots txt nofollow line, and a crawl-delay line are all inert as far as Googlebot is concerned. They parse as unknown rules and get discarded.

Other search engines never supported these directives in the first place. Bing has consistently pointed people to meta robots tags for indexing control, so a robots.txt noindex was only ever a Google-specific convention — which is part of why standardizing the protocol meant dropping it.

Nothing breaks visibly when this happens. That is the problem. There is no error, no Search Console warning, no signal at all — the page you believed was excluded simply keeps ranking. Teams often discover it during a content audit years later, when they find pages in the index that were “removed” in 2020.

The Logic Problem That Made It Unworkable Anyway

Set aside the deprecation for a moment. A noindex instruction inside robots.txt was always structurally odd, because robots.txt is the file that stops crawlers from fetching URLs. Combine a disallow and a noindex on the same path and you have given two contradictory instructions: do not look at this page, and also read the instruction on it.

This is the central trap. A URL blocked by robots.txt cannot be crawled. A page-level noindex signal can only be read by crawling. So blocking a URL does not merely fail to de-index it — it guarantees the de-index signal is never seen. If you want a page out of the index, Googlebot must be allowed to fetch it.

The Meta Robots Tag

For HTML pages, the replacement is a meta tag in the <head>:

<meta name="robots" content="noindex">

That applies to all crawlers that support the tag. To target Google only, use name="googlebot". Two combinations worth knowing:

  • content="noindex, follow" — drop the page from the index but keep passing link equity through its outbound links. Reasonable for paginated archives.
  • content="noindex, nofollow" — drop the page and ignore its links entirely. Use sparingly; it strands anything only linked from that page.

Note that noindex implies nofollow over time, because a de-indexed page eventually stops being crawled and its links stop being discovered. Do not rely on a noindexed page as a permanent crawl path.

The X-Robots-Tag Header

Meta tags only work in HTML. For PDFs, images, CSVs, and any non-HTML response, use the HTTP header instead:

X-Robots-Tag: noindex

In Apache, applied to all PDFs:

<FilesMatch "\.pdf$">
  Header set X-Robots-Tag "noindex"
</FilesMatch>

In Nginx, the equivalent is an add_header X-Robots-Tag "noindex"; inside a location block matching the extension. The header supports the same values as the meta tag, plus useful extras like unavailable_after with a date for time-limited content such as expiring job listings.

Multiple X-Robots-Tag headers can be sent on one response, and you can address a specific crawler by prefixing its token: X-Robots-Tag: googlebot: noindex. That precision is occasionally useful when you want a document out of Google but still discoverable by an internal or partner crawler.

The header approach also scales better on large sites. Setting it at the server or CDN level for an entire path pattern is one rule, versus editing thousands of templates.

The Correct Order of Operations

When you want a section of a site out of search and also want to stop wasting crawl on it, sequence matters:

  1. Make sure the URLs are not blocked in robots.txt. Remove the disallow if one exists.
  2. Serve noindex via meta tag or X-Robots-Tag on every affected URL.
  3. Confirm with Search Console’s URL Inspection that Google sees the tag on the live URL.
  4. Wait for recrawl and de-indexing. Small sections can clear in days; large ones take weeks.
  5. Only after the URLs have dropped out, add a robots.txt disallow if crawl efficiency still justifies it.

How long step 4 takes depends on how often Google crawls those URLs. High-authority pages that get fetched daily can drop within a week. Deep, rarely-crawled pages can take two months or more. Submitting the affected URLs in a temporary sitemap sometimes speeds up rediscovery, which sounds backwards but works — you are asking Google to come look at the page so it can read the instruction to leave.

Skip step 1 and nothing happens. Do step 5 too early and you freeze the pages in the index in exactly the half-state you were trying to escape.

Verifying That the Signal Is Being Seen

A noindex tag that Google never reads is no better than the robots.txt line you replaced. Three checks catch the common failures.

  • View source, not the rendered DOM. If your noindex tag is injected by client-side JavaScript, Google will only honor it after rendering — which happens later and less reliably than the initial fetch. Server-rendered tags are strictly better.
  • Check the response headers. Run curl -I https://example.com/page and look for a stray X-Robots-Tag. CDNs and security plugins sometimes add one you never asked for, and a header and meta tag that disagree resolve in favor of the more restrictive value.
  • Use URL Inspection on the live URL. It reports the indexing directive Google actually observed, which settles arguments that source-viewing cannot.

The header-versus-tag conflict is worth dwelling on. If a page carries <meta name="robots" content="index"> but the server sends X-Robots-Tag: noindex, the page is noindexed. I have watched teams spend a week debugging a template because nobody thought to check headers.

What About Nofollow?

A robots txt no follow directive has the same status as noindex — unsupported and ignored. Link-level control lives in the HTML: rel="nofollow" on individual anchors, or rel="sponsored" and rel="ugc" for paid and user-generated links respectively. Page-level control lives in the meta robots tag.

Since 2019, Google treats all three link attributes as hints rather than directives, meaning it may still choose to follow and consider a nofollowed link. If you need a URL genuinely excluded from crawling, robots.txt disallow remains the only reliable lever — with all the indexing caveats above.

Auditing What You Have Now

Open your robots.txt and search for noindex, nofollow, and crawl-delay. Every hit is a rule that has not functioned for years, and each one probably represents a page you believe is excluded but is not. Delete them, then check whether those URLs are in the index.

At scale this is a crawl job, not a manual one. SEO Rocket’s full-site crawls — quick scans of roughly 25 pages, or deep crawls verified past 900 — report indexability signals with concrete evidence per issue, including actual URLs, titles, and H1 text, so a template missing its noindex tag is a line item rather than a hunch. It does not edit your robots.txt or your server config, and it should not; those changes deserve a human. What it removes is the part where you find out six months late.

Keep reading