Crawl budget is the number of URLs a search engine crawler can and wants to fetch from a website within a given period. In SEO crawling, it is determined by two forces: the crawl capacity limit, meaning how much fetching the site’s servers can handle, and crawl demand, meaning how much the search engine wants to crawl the site based on its popularity and freshness.
Crawl budget matters because a page that is not crawled cannot be indexed or updated in search results. For most small and medium websites it is not a constraint at all. It becomes a real issue for very large sites, sites that change many pages daily, and sites that generate huge numbers of low-value URLs.
How crawl budget is determined
A web crawler such as Googlebot balances two goals: discovering and refreshing content, and not overloading the site. The crawl capacity limit rises when the server responds quickly and reliably, and falls when responses slow down or return server errors. The crawler adapts this limit continuously, so server health directly shapes crawl rate.
Crawl demand reflects how valuable recrawling seems. Popular URLs, URLs that change often and newly discovered sections attract more demand, while stale or duplicate URLs attract less. Site-wide events such as a domain migration can temporarily raise demand as the search engine reprocesses many addresses.
| Component | Driven by | Raises crawling when | Lowers crawling when |
|---|---|---|---|
| Crawl capacity limit | Server performance and errors | Responses are fast and stable | Timeouts, 5xx errors, 429 responses |
| Crawl demand | Perceived value and freshness | Pages are popular and updated | Content is duplicate, thin or unchanged |
When crawl budget actually matters
Crawl budget is a scale problem. A site with a few thousand well-linked pages is usually crawled thoroughly, and optimizing budget brings little benefit. The issue appears when the number of crawlable URLs is far larger than the number of valuable pages, or when content changes faster than the crawler revisits it.
| Site type | Typical URL scale | Crawl budget relevance | Main risk |
|---|---|---|---|
| Small business site | Hundreds to a few thousand | Low | Indexing issues have other causes |
| Content publisher | Tens of thousands, frequent updates | Medium | Slow pickup of fresh articles |
| Large e-commerce site | Hundreds of thousands to millions | High | Faceted URLs crowd out products |
| Classifieds or marketplace | Millions, short-lived listings | Very high | Expired pages consume crawling |
| JavaScript-heavy application | Varies | Medium to high | Rendering costs delay processing |
A reliable sign of a crawl budget problem is a growing set of URLs reported as discovered but not yet crawled, combined with important pages that take weeks to appear or update in search results.
What wastes crawl budget
Most crawl budget problems come from URL inflation, where a site exposes many addresses that lead to duplicate or low-value content. Every such fetch is a request that could have gone to a page that matters.
| Source of waste | Example | Recommended fix |
|---|---|---|
| Faceted navigation | Color, size and sort combinations | Block parameter patterns in robots.txt, limit crawlable facets |
| Session and tracking parameters | IDs appended to URLs | Remove from internal links, use clean URLs |
| Infinite spaces | Endless calendars or search pages | Disallow the pattern, cap pagination |
| Redirect chains | Several hops before the final URL | Link directly to the final destination |
| Soft 404 pages | Empty listings returning status 200 | Return a real 404 or 410 status |
| Duplicate content | HTTP and HTTPS or trailing-slash variants | Consolidate with redirects and canonicals |
A canonical tag consolidates indexing signals but does not stop crawling of the duplicate URL. Likewise, a noindex directive must be crawled to be seen. To save crawling itself, a URL pattern must be disallowed in robots.txt or removed from internal links.
How to optimize crawl budget
Optimization means making it easy to reach valuable URLs and hard to reach worthless ones, while keeping the server fast enough to support a high crawl rate.
- Improve server response time and eliminate recurring 5xx errors, since capacity rises with reliability.
- Keep XML sitemaps clean, listing only canonical, indexable URLs with accurate last-modified dates.
- Strengthen internal linking so that important pages sit within a few clicks of the home page.
- Return correct status codes: 404 or 410 for removed content, 304 for unchanged resources where supported.
- Analyze server logs to see which URLs crawlers actually request and how often.
Server log analysis is the most reliable diagnostic, because it shows real crawler behavior rather than estimates. Comparing crawled URLs with the list of valuable pages reveals both waste and neglected sections.
Crawl budget compared with related concepts
Crawl budget is frequently confused with neighboring ideas in crawling and indexing. Being crawled does not guarantee being indexed, and blocking crawling does not guarantee removal from results.
| Concept | What it controls | Relation to crawl budget |
|---|---|---|
| Crawl rate | Requests per second to the server | The momentary speed within the budget |
| Robots.txt | Which paths may be fetched | Main tool to stop wasted crawling |
| Crawl error | Failed fetch attempts | Errors lower capacity and waste requests |
| Index coverage | Which crawled pages are stored | Separate quality decision after crawling |
| Rendering | Executing JavaScript to see content | Adds processing cost per page |
Explained without jargon
Imagine a librarian who visits a huge bookstore every day but only has time to check a limited number of shelves. The bookstore can help by keeping aisles clear and not filling shelves with endless copies of the same book.
Crawl budget is that limited visiting time for a search engine. If it is wasted on useless pages, important new pages wait longer before anyone can find them in search.
A typical use case
An online fashion retailer with about 80,000 products notices that new collections take weeks to appear in search results. Server log analysis shows that most crawler requests go to filter combinations such as color, size, price and sort order, which generate millions of near-duplicate URLs.
The team blocks sorting and multi-filter parameters in robots.txt, removes them from internal links, keeps only a few valuable category filters crawlable, and cleans the XML sitemap. Within a few weeks, logs show crawler activity shifting toward product and category pages, and new products are indexed much faster after launch.