The X-Robots-Tag is an HTTP response header that tells search engine crawlers how to index and display a resource, using the same directives as the meta robots tag but delivered by the server instead of the HTML. In SEO and especially in indexing, it is the standard way to apply rules such as noindex, nofollow or noarchive to non-HTML files like PDFs, images, videos and feeds, and to apply them across many URLs at once.
Because the directive lives in the response header, it can be set in web server configuration, a CDN or application code without editing any page template. That flexibility makes the X-Robots-Tag powerful for technical teams and also risky: one misconfigured rule can remove an entire directory from search results.
How the X-Robots-Tag header works
The X-Robots-Tag works by adding a header line to the HTTP response that a crawler reads before it processes the body. A response might contain X-Robots-Tag: noindex, nofollow, which tells compliant crawlers not to index the resource and not to follow its links. The directive is honored only when the crawler is allowed to fetch the URL, because a blocked URL never returns headers to read.
Directives can be combined in one comma-separated header or split across several headers. They can also target a specific user agent, as in X-Robots-Tag: googlebot: noindex, which applies only to Googlebot while other crawlers follow untargeted rules. When directives conflict, search engines generally apply the most restrictive rule.
The table below lists the directives most often used in the header and their effect.
| Directive | Effect | Typical use |
|---|---|---|
| noindex | Excludes the resource from the index | Internal PDFs, staging files |
| nofollow | Links in the resource are not followed | Untrusted or generated documents |
| none | Equivalent to noindex and nofollow | Fully private resources |
| noarchive | No cached copy is shown | Frequently changing price files |
| nosnippet | No text snippet in search results | Licensed or sensitive content |
| noimageindex | Images on the page are not indexed | Pages embedding licensed images |
| unavailable_after | Removal after a stated date and time | Expiring offers and events |
| max-snippet:[n] | Limits snippet length to n characters | Publisher snippet control |
X-Robots-Tag compared with meta robots and robots.txt
The X-Robots-Tag, the meta robots tag and robots.txt control different stages of how search engines handle a URL. Robots.txt governs crawling, while the two robots directives govern indexing and presentation. Confusing these layers is one of the most common causes of indexing errors.
| Mechanism | Where it lives | Controls | File types |
|---|---|---|---|
| X-Robots-Tag | HTTP response header | Indexing and snippets | Any file type |
| Meta robots tag | HTML head of a page | Indexing and snippets | HTML only |
| Robots.txt | Text file at the site root | Crawling access | Any path pattern |
The critical interaction is that robots.txt blocking prevents noindex from working. If a directory is disallowed in robots.txt, crawlers never request its files, never see the X-Robots-Tag and may still index the bare URLs when other sites link to them. To remove a resource from the index, the URL must stay crawlable until the noindex is processed.
Between the two indexing mechanisms, the meta robots tag is simpler for individual HTML pages edited in a content management system, while the X-Robots-Tag is the only option for non-HTML files and the more efficient choice for pattern-based rules across many URLs.
Configuring the X-Robots-Tag on servers and CDNs
The X-Robots-Tag is configured wherever the HTTP response is assembled: in the web server, a reverse proxy or CDN, or the application layer. The choice depends on who owns the infrastructure and how precisely the rule must target URLs.
On Apache, the Header set directive inside a FilesMatch block can add noindex to every PDF. On Nginx, an add_header directive in a location block does the same for a path or file extension. Application frameworks and edge functions can set the header conditionally, for example only on filtered search pages or only for logged-in previews.
| Layer | Strength | Main risk |
|---|---|---|
| Web server configuration | Fast, pattern-based rules for file types | Broad patterns catch too many URLs |
| CDN or edge rules | Central control across origins | Rules hidden from developers |
| Application code | Precise, data-driven conditions | Bugs ship with releases |
| Staging environment | Blocks indexing of test hosts | Header copied to production |
- Verify the live header with a header inspection tool or the URL Inspection tool, not only the configuration file.
- Document every rule with its purpose and owner so later teams do not remove or duplicate it.
- Add monitoring for unexpected noindex headers on key templates after each deployment.
Practical uses in indexing management
The X-Robots-Tag is most valuable for controlling index bloat and keeping low-value or duplicate resources out of search results. Typical cases include printable PDF copies of HTML pages, internal search result pages, feeds, faceted URLs and attachment pages generated automatically by a content management system.
It also handles duplicate content in non-HTML formats. A PDF version of a report cannot carry a canonical tag in its markup, but the server can send an HTTP Link header with rel="canonical" or an X-Robots-Tag noindex, depending on whether the PDF should consolidate signals to the HTML page or simply disappear from results.
Removal is not instant. The directive takes effect only after the URL is recrawled, which can take days or weeks for rarely visited files. For urgent cases, a temporary removal request in the search console hides the URL while the noindex is processed permanently.
Common X-Robots-Tag mistakes
Most X-Robots-Tag problems come from rules that are broader than intended or that survive longer than planned. Because headers are invisible in the page source, these errors often go unnoticed until traffic drops.
| Symptom | Likely cause | Fix |
|---|---|---|
| Whole site drops from the index | Staging noindex header deployed live | Remove the header and request recrawl |
| PDFs still appear in results | PDF path blocked in robots.txt | Allow crawling until noindex is seen |
| Images vanish from image search | noindex applied to image extensions | Narrow the file pattern |
| Directive ignored by one crawler | Header targets another user agent | Add an untargeted rule |
| Pages show as excluded by noindex | CDN rule matched the wrong path | Audit edge rules and test headers |
A further misconception is that noindex also stops crawling. Search engines continue to request noindexed URLs, although usually less often, so the header is not a tool for saving crawl budget on its own.
Understanding it simply
Every time a website sends a file to a visitor, it attaches a small invisible label with technical information. The X-Robots-Tag is one of those labels, written for search engines. It can say, for example, do not show this file in search results.
The advantage is that the label works on any kind of file, not only on web pages, so a company can hide hundreds of internal PDFs with a single setting. The danger is the same: one wrong setting can hide far too much, so these labels need to be checked carefully.
Applied in practice
A university publishes about 8,000 PDF files, including outdated course handbooks, committee minutes and printable copies of web pages. Search results for program names were showing old handbooks instead of the current program pages, and the index coverage report listed thousands of low-value PDFs.
The web team added an Nginx rule that sends X-Robots-Tag: noindex for PDFs in the archive and minutes directories, while current brochures stayed indexable. Printable copies received a canonical Link header pointing to their HTML pages. The team confirmed that robots.txt did not block those paths, so crawlers could read the new headers. Within two months the number of indexed PDFs fell by roughly 70 percent, and searches for program names returned the current program pages first.