Robots.txt

Robots.txt is a plain-text file placed at the root of a host that tells web crawlers which URL paths they may request. It implements the Robots Exclusion Protocol, which was formalized as RFC 9309 in 2022 after decades of informal use. In crawling and SEO, robots.txt is the first file a well-behaved crawler such as Googlebot fetches before requesting pages from a site.

Robots.txt controls crawling, not indexing. Blocking a URL prevents compliant crawlers from reading its content, but the URL itself can still appear in search results if other pages link to it. It is also a voluntary convention, not a security mechanism: reputable crawlers obey it, while malicious bots can ignore it entirely.

How robots.txt works

A crawler requests /robots.txt from each protocol, host and port combination separately, so the main www host and a separate shop subdomain each need their own file. The file consists of groups: each group begins with one or more User-agent lines naming the crawlers it applies to, followed by Allow and Disallow rules that match URL paths.

A crawler uses the single most specific group that matches its user-agent token and ignores the others; a group for * applies only when no more specific group exists. Within the chosen group, the longest matching rule wins, and when an Allow and a Disallow rule are equally long, major crawlers apply the less restrictive Allow.

Pattern matching and technical limits

Rules match from the start of the URL path, case-sensitively. Two special characters extend matching: the asterisk * matches any sequence of characters, and the dollar sign $ anchors a rule to the end of the URL. An empty Disallow: line means that nothing is blocked.

Crawlers also apply practical limits. Google processes only the first 500 KiB of the file, and crawlers commonly cache robots.txt for up to about 24 hours, so changes do not take effect instantly. The HTTP status of the file matters as well, as the following table shows.

Robots.txt compared with other crawl and index controls

SEO specialists choose among several mechanisms that look similar but act at different stages of the search pipeline. Confusing them is one of the most common technical SEO errors.

A crucial interaction follows from this: a noindex directive on a page blocked by robots.txt is never seen, because the crawler is not allowed to fetch the page. To remove a page from the index, it must stay crawlable until the noindex has been processed.

Robots.txt and crawl budget

On large sites, robots.txt is a primary tool for managing crawl budget, the amount of crawling a search engine is willing and able to spend on a host. Blocking faceted navigation, internal search results, session parameters and infinite calendars keeps crawlers focused on valuable pages. Small sites with a few hundred URLs rarely need such tuning.

Blocking is not always the best crawl budget tool. Parameter URLs that are already indexed may be better handled with canonical links or noindex first, and blocked only afterward, because a sudden block freezes the existing index entries in place. Server log analysis is the most reliable way to see which areas crawlers actually spend time on before writing new rules.

Robots.txt has also become the main way to express preferences toward AI crawlers. Many operators publish separate groups for crawlers that collect training data, using each company’s documented user-agent token, while keeping search crawlers allowed. These rules remain voluntary and depend on each operator honoring them.

Common robots.txt mistakes

A single wrong line in robots.txt can remove an entire site from organic search, which makes it one of the highest-risk files in technical SEO.

• Copying a staging file to production, where Disallow: / blocks the whole live site.
• Blocking CSS and JavaScript, which prevents search engines from rendering pages correctly.
• Using robots.txt to hide sensitive URLs, which publicly lists exactly the paths an attacker might want.
• Expecting Disallow to deindex pages that are already indexed or linked from elsewhere.
• Relying on unsupported directives such as noindex inside robots.txt, which Google does not support.
• Skipping validation: every change should be tested against sample URLs with a robots.txt testing tool before deployment.

Put simply

Robots.txt is like a sign at the entrance of a building telling visiting robots which rooms they may enter. Search engine robots read the sign before walking around the website and politely stay out of the rooms marked as closed.

The sign does not lock any doors. It only works because well-behaved robots choose to follow it, so anything truly private still needs a real lock, such as a password.

In practice

An online fashion retailer with about 50,000 products notices that search engines spend most of their crawling on filter combinations such as color, size and sort order, generating millions of near-duplicate URLs, while new products take weeks to appear in results.

The SEO team adds rules such as Disallow: /*?sort= and blocks internal search pages, keeps product and category pages open, and lists the XML sitemap in robots.txt. After testing the rules against sample URLs, they deploy the file. Within weeks, server logs show crawling concentrated on product and category pages, and new products are discovered and indexed noticeably faster.