X-Robots-Tag

On this page 7

The X-Robots-Tag is an HTTP response header that tells search engine crawlers how to index and display a resource, using the same directives as the meta robots tag but delivered by the server instead of the HTML. In SEO and especially in indexing, it is the standard way to apply rules such as noindex, nofollow or noarchive to non-HTML files like PDFs, images, videos and feeds, and to apply them across many URLs at once.

Because the directive lives in the response header, it can be set in web server configuration, a CDN or application code without editing any page template. That flexibility makes the X-Robots-Tag powerful for technical teams and also risky: one misconfigured rule can remove an entire directory from search results.

How the X-Robots-Tag header works

The X-Robots-Tag works by adding a header line to the HTTP response that a crawler reads before it processes the body. A response might contain X-Robots-Tag: noindex, nofollow, which tells compliant crawlers not to index the resource and not to follow its links. The directive is honored only when the crawler is allowed to fetch the URL, because a blocked URL never returns headers to read.

Directives can be combined in one comma-separated header or split across several headers. They can also target a specific user agent, as in X-Robots-Tag: googlebot: noindex, which applies only to Googlebot while other crawlers follow untargeted rules. When directives conflict, search engines generally apply the most restrictive rule.

The table below lists the directives most often used in the header and their effect.

DirectiveEffectTypical use
noindexExcludes the resource from the indexInternal PDFs, staging files
nofollowLinks in the resource are not followedUntrusted or generated documents
noneEquivalent to noindex and nofollowFully private resources
noarchiveNo cached copy is shownFrequently changing price files
nosnippetNo text snippet in search resultsLicensed or sensitive content
noimageindexImages on the page are not indexedPages embedding licensed images
unavailable_afterRemoval after a stated date and timeExpiring offers and events
max-snippet:[n]Limits snippet length to n charactersPublisher snippet control
Common X-Robots-Tag directives

X-Robots-Tag compared with meta robots and robots.txt

The X-Robots-Tag, the meta robots tag and robots.txt control different stages of how search engines handle a URL. Robots.txt governs crawling, while the two robots directives govern indexing and presentation. Confusing these layers is one of the most common causes of indexing errors.

MechanismWhere it livesControlsFile types
X-Robots-TagHTTP response headerIndexing and snippetsAny file type
Meta robots tagHTML head of a pageIndexing and snippetsHTML only
Robots.txtText file at the site rootCrawling accessAny path pattern
Three ways to control crawler behavior

The critical interaction is that robots.txt blocking prevents noindex from working. If a directory is disallowed in robots.txt, crawlers never request its files, never see the X-Robots-Tag and may still index the bare URLs when other sites link to them. To remove a resource from the index, the URL must stay crawlable until the noindex is processed.

Between the two indexing mechanisms, the meta robots tag is simpler for individual HTML pages edited in a content management system, while the X-Robots-Tag is the only option for non-HTML files and the more efficient choice for pattern-based rules across many URLs.

Configuring the X-Robots-Tag on servers and CDNs

The X-Robots-Tag is configured wherever the HTTP response is assembled: in the web server, a reverse proxy or CDN, or the application layer. The choice depends on who owns the infrastructure and how precisely the rule must target URLs.

On Apache, the Header set directive inside a FilesMatch block can add noindex to every PDF. On Nginx, an add_header directive in a location block does the same for a path or file extension. Application frameworks and edge functions can set the header conditionally, for example only on filtered search pages or only for logged-in previews.

LayerStrengthMain risk
Web server configurationFast, pattern-based rules for file typesBroad patterns catch too many URLs
CDN or edge rulesCentral control across originsRules hidden from developers
Application codePrecise, data-driven conditionsBugs ship with releases
Staging environmentBlocks indexing of test hostsHeader copied to production
Where to set the header and when
  • Verify the live header with a header inspection tool or the URL Inspection tool, not only the configuration file.
  • Document every rule with its purpose and owner so later teams do not remove or duplicate it.
  • Add monitoring for unexpected noindex headers on key templates after each deployment.

Practical uses in indexing management

The X-Robots-Tag is most valuable for controlling index bloat and keeping low-value or duplicate resources out of search results. Typical cases include printable PDF copies of HTML pages, internal search result pages, feeds, faceted URLs and attachment pages generated automatically by a content management system.

It also handles duplicate content in non-HTML formats. A PDF version of a report cannot carry a canonical tag in its markup, but the server can send an HTTP Link header with rel="canonical" or an X-Robots-Tag noindex, depending on whether the PDF should consolidate signals to the HTML page or simply disappear from results.

Removal is not instant. The directive takes effect only after the URL is recrawled, which can take days or weeks for rarely visited files. For urgent cases, a temporary removal request in the search console hides the URL while the noindex is processed permanently.

Common X-Robots-Tag mistakes

Most X-Robots-Tag problems come from rules that are broader than intended or that survive longer than planned. Because headers are invisible in the page source, these errors often go unnoticed until traffic drops.

SymptomLikely causeFix
Whole site drops from the indexStaging noindex header deployed liveRemove the header and request recrawl
PDFs still appear in resultsPDF path blocked in robots.txtAllow crawling until noindex is seen
Images vanish from image searchnoindex applied to image extensionsNarrow the file pattern
Directive ignored by one crawlerHeader targets another user agentAdd an untargeted rule
Pages show as excluded by noindexCDN rule matched the wrong pathAudit edge rules and test headers
Symptoms, causes and fixes

A further misconception is that noindex also stops crawling. Search engines continue to request noindexed URLs, although usually less often, so the header is not a tool for saving crawl budget on its own.

Understanding it simply

Every time a website sends a file to a visitor, it attaches a small invisible label with technical information. The X-Robots-Tag is one of those labels, written for search engines. It can say, for example, do not show this file in search results.

The advantage is that the label works on any kind of file, not only on web pages, so a company can hide hundreds of internal PDFs with a single setting. The danger is the same: one wrong setting can hide far too much, so these labels need to be checked carefully.

Applied in practice

A university publishes about 8,000 PDF files, including outdated course handbooks, committee minutes and printable copies of web pages. Search results for program names were showing old handbooks instead of the current program pages, and the index coverage report listed thousands of low-value PDFs.

The web team added an Nginx rule that sends X-Robots-Tag: noindex for PDFs in the archive and minutes directories, while current brochures stayed indexable. Printable copies received a canonical Link header pointing to their HTML pages. The team confirmed that robots.txt did not block those paths, so crawlers could read the new headers. Within two months the number of indexed PDFs fell by roughly 70 percent, and searches for program names returned the current program pages first.