Noindex is a robots directive that instructs search engines not to include a page in their index, so the page does not appear in search results even though it remains accessible to visitors. In SEO and indexing, it is the standard, reliable method for keeping a crawlable page out of search, delivered either as a meta robots tag in the HTML or as an HTTP response header.
Noindex controls indexing, not crawling. A search engine must be able to fetch the page to read the directive, which is why noindex and robots.txt blocking are not interchangeable. Used correctly, noindex keeps thin, private or utility pages from diluting a site’s presence in search.
How the noindex directive is implemented
There are two supported ways to deliver noindex. The meta robots tag sits in the head of an HTML page, written as meta name="robots" content="noindex". The X-Robots-Tag HTTP header carries the same directive in the server response, as X-Robots-Tag: noindex, and works for any file type.
| Method | Where it lives | Works for | Typical use |
|---|---|---|---|
| Meta robots tag | HTML head section | HTML pages only | Individual pages managed in a CMS |
| Crawler-specific meta tag | HTML head, named for one bot | HTML pages, one crawler | Excluding a page from one search engine |
| X-Robots-Tag header | HTTP response header | Any file, including PDFs and images | Bulk rules set on the server |
| Noindex in robots.txt | robots.txt file | Not supported by major engines | Should not be used |
Google stopped honoring noindex rules inside robots.txt in 2019, so that approach no longer works and leaves pages indexable. Directives can also be combined, for example noindex with nofollow, or narrowed with a user agent name to address a single crawler.
What happens after a search engine sees noindex
When a crawler fetches a page and finds noindex, the search engine drops the page from its index on the next processing, or never adds it if it was new. Removal therefore depends on when the page is recrawled, which can take days for important pages and weeks or longer for rarely visited ones.
Two further effects matter for site architecture. First, links on a noindexed page may still be followed at first, but search engines have indicated that a page kept noindexed for a long time is eventually treated like noindex, nofollow, so it stops passing signals to the pages it links to. Second, a noindexed page cannot rank and accumulates no visibility, regardless of its quality or backlinks.
For pages rendered with JavaScript, the directive should be present in the initial HTML response. If the original HTML contains noindex, the rendering step may be skipped, so removing noindex later with JavaScript is unreliable.
Noindex compared with other indexing controls
Noindex is one of several tools for managing what appears in search, and each one solves a different problem. Choosing the wrong one is the most common source of indexing errors.
| Control | Prevents crawling | Prevents indexing | Consolidates signals |
|---|---|---|---|
| Noindex | No | Yes | No |
| Robots.txt disallow | Yes | Not reliably | No |
| Canonical URL | No | Usually, as a hint | Yes, toward the canonical |
| 301 redirect | Effectively, for the old URL | Yes, for the old URL | Yes, toward the target |
| Password protection | Yes | Yes | No |
The key distinction is between duplicate content and unwanted content. For duplicates, a canonical URL or a redirect passes signals to the preferred version. For pages that should not be found at all, noindex or authentication is the right tool. Combining noindex and a canonical tag pointing elsewhere sends conflicting signals and should be avoided.
When to use noindex
Noindex suits pages that serve users on the site but add nothing to search results. The decision table below lists typical cases.
| Page type | Recommendation | Reason |
|---|---|---|
| Internal search results | Noindex | Endless thin variations of existing content |
| Thank-you and confirmation pages | Noindex | No value as a search landing page |
| Staging or test pages | Authentication, plus noindex as backup | Noindex alone does not keep them private |
| Filtered category variants | Canonical or noindex, depending on value | Some filters deserve their own ranking page |
| Product page variants with near-identical content | Canonical URL | Signals should consolidate, not disappear |
| Expired content with a successor | 301 redirect | Preserves link value for the replacement |
Noindex is not a security control. A noindexed page can still be opened by anyone who has the URL, so confidential material must sit behind authentication.
Common noindex mistakes and how to detect them
Noindex errors can remove important pages from search silently, so they deserve routine monitoring in index coverage reports and in crawls of the live site.
- Staging noindex left on after launch, which removes an entire site from search.
- Blocking a noindexed page in robots.txt, so crawlers never see the directive and the URL may stay indexed without content.
- CMS or plugin settings applying noindex to whole sections such as categories or tags by accident.
- Noindex on paginated series that hides deep products or articles from discovery over time.
- Conflicting directives, such as noindex combined with a canonical tag or listing the URL in an XML sitemap.
Index coverage tools typically report such URLs as excluded by noindex. Reviewing that list after every release, and comparing it with the pages intended for search, catches most problems before traffic is lost.
Put simply
Noindex is a polite note on a web page that tells search engines not to list it. People can still visit the page, but it will not show up when someone searches.
It is like a shop keeping its storeroom unlisted in the mall directory. The room exists and staff use it, but shoppers are not sent there. The search engine must still be able to walk in and read the note, or it cannot follow the request.
In practice
A large recipe site notices that thousands of internal search result pages appear in the index, each one a thin list of recipes for queries such as easy chicken dinner. Index coverage shows these pages competing with the site’s own curated collection pages and diluting crawl activity.
The team adds an X-Robots-Tag noindex header to every URL under the internal search path and keeps those URLs crawlable in robots.txt so the directive can be read. Over the following weeks, the search pages drop out of the index as they are recrawled, and the curated collections become the pages that rank. The team also removes the search URLs from the XML sitemap to avoid sending conflicting signals.