The meta robots tag is a small line in your page’s <head> that gives search engines page-level instructions, most importantly, whether to index the page and whether to follow its links. Understanding it is essential technical SEO: used correctly it keeps low-value pages out of search; used carelessly it can accidentally hide pages you want ranked. This guide explains the directives, when to use each, and how the meta robots tag differs from robots.txt.
What the meta robots tag is
The meta robots tag is an HTML <meta> element placed in the head of a page, for example <meta name="robots" content="noindex, follow">. It tells crawlers how to handle that specific page. Unlike robots.txt, which controls whether a bot may crawl a URL, the meta robots tag controls what happens once the page is crawled, chiefly whether it may appear in the search index. That distinction is the single most important thing to understand about it.
The common directives
A handful of values cover almost every case:
| Directive | Meaning |
|---|---|
| index / noindex | Allow or prevent the page appearing in search |
| follow / nofollow | Follow or ignore the links on the page |
| noarchive | Don’t show a cached copy |
| nosnippet | Don’t show a text snippet in results |
| max-snippet / max-image-preview | Limit snippet or image preview size |
The default for any page is index, follow, so you only need the tag when you want something other than the default. The most-used non-default is noindex.
When to use noindex
Noindex is for pages that need to exist and be crawlable but should not appear in search results. Typical cases: thank-you and confirmation pages, internal search-results pages, thin tag or filter pages, staging or duplicate pages, and admin or account pages. A common and sensible combination is noindex, follow, keep the page out of the index but still let its links pass through to other pages. The key point: to reliably keep a page out of search, use noindex, not robots.txt.
The robots.txt trap
This is the mistake worth burning into memory: robots.txt and noindex are not interchangeable. Robots.txt blocks crawling; the meta robots tag controls indexing. Here is the trap, if you block a page in robots.txt, Google may never crawl it, which means it never sees your noindex tag, so the page can still end up indexed (as a bare URL) from external links. To keep a page out of the index, you must let it be crawled and put a noindex tag on it. Blocking it in robots.txt does the opposite of what people expect. Whenever you want something gone from search, reach for meta robots noindex, and make sure the page is not also blocked from crawling.
Meta robots vs the X-Robots-Tag
The same directives can be delivered two ways. The meta robots tag works for HTML pages, where you can add something to the <head>. But for non-HTML files, PDFs, images, other documents, there is no head to put a meta tag in. For those, the same instructions are sent as an HTTP response header called the X-Robots-Tag, configured on the server. The directives are identical (noindex, nofollow and the rest); only the delivery differs. Knowing both exist means you are never stuck: use the meta tag for pages and the X-Robots-Tag header when you need to control indexing of a file that has no HTML head.
Generate your meta robots tag
Getting the directive combination and syntax right is easy with a generator. Our free meta robots generator builds the exact tag from the options you choose, index/noindex, follow/nofollow, and the snippet controls, ready to paste into your head. To manage crawling (the other half of the picture), see our robots.txt generator and the guide on creating a robots.txt file.
The takeaway
The meta robots tag gives page-level indexing instructions, most importantly noindex to keep a page out of search and follow/nofollow for its links. The critical rule: to remove a page from the index, use noindex and keep the page crawlable, do not block it in robots.txt, or Google may never see the noindex. For non-HTML files, use the X-Robots-Tag header instead. Generate the right tag with our meta robots generator, and explore the rest of our free SEO tools.