Crawl rate is the number of requests a search engine crawler makes to a website in a given period, typically expressed as requests per second or per day. In crawling and SEO, it describes how fast bots such as Googlebot or Bingbot fetch URLs from a host, and it is governed by a balance between what the server can handle and how much the search engine wants to crawl.
Crawl rate is one of the two ingredients of crawl budget. It sets the upper speed limit, while crawl demand decides which URLs actually fill that capacity. A healthy crawl rate means new and changed pages are discovered quickly without overloading the server.
How search engines set the crawl rate
Search engines set the crawl rate automatically through a feedback loop based on server responses. The crawler starts at a cautious level, raises the number of parallel connections when the host responds quickly and without errors, and slows down immediately when responses become slow or fail.
Google describes this upper bound as the crawl capacity limit: the maximum number of simultaneous connections and the delay between fetches that Googlebot will use for a host. The limit rises with fast, stable responses and falls with timeouts, 5xx errors and 429 responses. Server capacity alone does not force more crawling, however; without sufficient demand the crawler stays below the limit.
The table lists the server signals that most strongly influence the crawl rate.
| Signal | Effect on crawl rate | Typical cause |
|---|---|---|
| Fast response times | Gradual increase | Efficient hosting, caching, CDN |
| Slow responses | Gradual decrease | Overloaded database, heavy pages |
| 500 and 503 errors | Sharp, temporary decrease | Server failure or maintenance |
| 429 Too Many Requests | Sharp, temporary decrease | Rate limiting by a firewall or CDN |
| Connection timeouts | Sharp decrease | Network or capacity problems |
| Stable 200 and 304 responses | Allows higher capacity | Healthy, cache-aware server |
Crawl rate compared with related crawling terms
Crawl rate is often confused with crawl budget, crawl demand and crawl delay, although each describes a different aspect of the crawling process. Clear distinctions matter when diagnosing why pages are discovered slowly.
| Term | Definition | Relation to crawl rate |
|---|---|---|
| Crawl rate | Speed of requests to a host | The speed itself |
| Crawl capacity limit | Maximum safe speed for a host | Ceiling for the crawl rate |
| Crawl demand | How much the engine wants to crawl | Decides how much of the capacity is used |
| Crawl budget | URLs the engine can and wants to crawl | Combines capacity and demand |
| Crawl delay | Directive asking for a pause between requests | Manual request to lower the rate |
| Crawl depth | Clicks from the start page to a URL | Affects which URLs are reached, not speed |
Support for manual controls differs between engines. Bingbot honors the crawl-delay directive in robots.txt and offers crawl control settings in its webmaster tools, whereas Googlebot ignores crawl-delay and, since early 2024, no longer offers a manual crawl rate setting in Search Console.
Measuring crawl rate
Crawl rate is measured from server logs and webmaster tools, which show how many requests each crawler made and how the server responded. Log file analysis gives the most precise view because it records every verified bot request with timestamp, URL and status code.
| Source | What it shows | Limitation |
|---|---|---|
| Server access logs | Every request by user agent, status and time | Needs verification of genuine bots |
| Crawl Stats report | Daily requests, download size, response time, host status | Aggregated, about 90 days of history |
| Bing Webmaster Tools | Crawl requests and errors over time | Covers only Bingbot |
| CDN analytics | Bot traffic at the edge | May miss requests served from origin |
Absolute numbers vary enormously. A small business site may receive a few hundred crawler requests per day, while a large marketplace can receive millions. The useful metric is therefore trend and proportion: whether important URLs are crawled often enough and whether crawling follows content changes.
Managing crawl rate in practice
Crawl rate is managed indirectly by improving server performance and reducing wasted requests, rather than by forcing a number. The goal is a rate that the server sustains comfortably and that is spent on valuable URLs.
| Situation | Recommended action | Reason |
|---|---|---|
| Crawler overloads the server | Return 503 or 429 temporarily, then add capacity | Signals the crawler to slow down safely |
| New pages discovered slowly | Improve internal links and XML sitemaps | Raises crawl demand for important URLs |
| Requests wasted on parameters | Consolidate URLs, block crawler traps | Frees capacity for valuable pages |
| Slow average response time | Add caching and optimize the backend | Allows a higher capacity limit |
| Planned maintenance | Serve 503 with a Retry-After header | Avoids errors being treated as permanent |
Error responses should stay temporary. Serving 503 or 429 for more than a day or two can cause the crawler to treat URLs as unavailable, and prolonged errors may lead to pages dropping from the index.
Common mistakes with crawl rate
Most crawl rate mistakes come from treating crawling as a goal in itself instead of a means of keeping the index current. The following misunderstandings appear repeatedly in technical audits.
- Treating a higher rate as a ranking boost. More crawling does not improve rankings; it only speeds discovery.
- Blocking crawlers to save resources. A robots.txt disallow stops crawling but can leave URLs indexed without content.
- Relying on crawl-delay for Google. Googlebot ignores the directive entirely.
- Ignoring fake bots. User agents can be spoofed, so verify crawlers by reverse DNS before analyzing rates.
- Worrying on small sites. Sites with a few thousand URLs rarely face real crawl rate constraints.
The idea in brief
Crawl rate is how quickly a search engine’s robot reads a website. The robot visits page after page to see what is new, and crawl rate is the pace of those visits.
Imagine a librarian checking a bookstore for new titles. If the shop is spacious and well organized, the librarian moves fast; if the aisles are crowded, the librarian slows down so as not to get in the way. Search engines behave the same way: a fast, reliable website gets visited faster, while a struggling one is visited gently.
A practical scenario
A regional news publisher notices in its server logs that Googlebot requests fell from about 40,000 to 12,000 per day within a week, and breaking stories now take hours to appear in search. The Crawl Stats report shows average response time rising above 1,500 milliseconds and a spike in 5xx errors after a new comment plugin was installed.
The developers move comments to asynchronous loading and enable page caching. Within two weeks, average response time falls below 300 milliseconds, errors disappear and crawl requests return to their earlier level. New articles are again crawled within minutes of publication.