A crawl error is a failure that occurs when a search engine crawler tries to fetch a website or a URL and cannot retrieve it successfully. In crawling, the term covers problems at the level of the whole site, such as DNS or server failures, and problems with individual URLs, such as 404 responses, redirect loops or blocked resources.
Crawl errors matter because a page that cannot be fetched cannot be indexed or refreshed, and widespread errors make crawlers slow down their visits. Not every reported error is a problem, however: a 404 for a page that was intentionally removed is the correct response, and the skill lies in separating real defects from expected behavior.
How a crawl error arises in the fetch process
Every crawl request passes through a fixed sequence of steps, and a crawl error is a failure at one of them. The crawler resolves the hostname through DNS, opens a connection, checks robots.txt, requests the URL and finally interprets the HTTP status code and the returned content.
| Stage | Typical failure | Effect |
|---|---|---|
| DNS resolution | Hostname does not resolve or times out | Entire site unreachable |
| Connection | Timeout, refused connection, TLS error | Requests abandoned |
| Robots.txt fetch | File returns 5xx or times out | Crawler may pause the whole site |
| URL request | 4xx or 5xx status | URL not fetched or dropped |
| Redirect handling | Loop, chain too long, invalid target | Final page never reached |
| Rendering | Blocked scripts or resources | Content not seen as users see it |
The robots.txt stage deserves special attention. Googlebot treats a 404 for robots.txt as permission to crawl everything, but a persistent 5xx response for robots.txt can make it stop crawling the site until the file becomes available again.
Site-level and URL-level crawl errors
Crawl errors fall into two classes with very different urgency. Site-level errors block access to everything and need immediate attention; URL-level errors affect individual addresses and are prioritized by the value of the affected pages.
| Error | Meaning | Usual fix |
|---|---|---|
| 404 not found | URL does not exist | Redirect if a replacement exists, otherwise leave it |
| 410 gone | URL removed permanently | Keep for deliberate removals |
| Soft 404 | Empty or error page returns 200 | Return a real 404 or add content |
| 5xx server error | Server failed to answer | Fix capacity, timeouts or application bugs |
| 403 forbidden | Access denied to the crawler | Check firewall, bot rules and authentication |
| Redirect error | Loop, chain or broken target | Point directly to the final URL |
| Blocked by robots.txt | Crawling disallowed | Allow if the URL should be crawled |
404 errors are normal on any living website and do not harm the rest of the site. They become a problem only when valuable URLs with links or traffic return 404 by mistake, or when internal links keep pointing to dead pages and waste crawl budget.
How crawl errors affect crawling and indexing
Crawlers adapt their behavior to server health. Frequent 5xx responses, 429 responses and timeouts signal overload, so the crawler lowers its crawl rate, which delays discovery of new pages and refreshes of existing ones.
If a URL keeps returning errors, search engines eventually drop it from the index. Short outages of a few hours rarely have lasting effects, but errors lasting several days can remove important pages from results until they are recrawled successfully.
The table below separates crawl errors from neighboring crawling concepts that are often confused with them.
| Concept | What it describes | Difference from a crawl error |
|---|---|---|
| Crawl budget | How many URLs a crawler will fetch | Errors waste budget but are not the budget |
| Crawl rate | Speed of requests to a host | Reduced as a response to errors |
| Crawler trap | Endless generated URLs | URLs often fetch fine but have no value |
| Redirect chain | Several hops before the final URL | Becomes an error only when too long or looping |
| Orphan page | Page with no internal links | Fetchable, just hard to discover |
Detecting and diagnosing crawl errors
Reliable diagnosis combines search engine reports, server logs and a site crawler. Each source sees a different part of the picture, so relying on one alone misses errors or exaggerates them.
- Page indexing and crawl stats reports in Google Search Console show errors as Google experienced them, with sample URLs and host status.
- Server log file analysis shows every real crawler request, its status code and response time.
- Desktop or cloud site crawlers reproduce errors on demand and reveal the internal links pointing to broken URLs.
- Uptime and DNS monitoring catches site-level outages before crawlers report them days later.
A useful rule is to verify every reported error with a live fetch before fixing it. Many errors are transient, and others come from firewalls or bot protection that block crawlers while serving human visitors normally.
Prioritizing fixes and avoiding common mistakes
Crawl errors should be fixed in order of impact on valuable URLs, not in order of count. A single robots.txt outage outweighs thousands of 404s from deleted spam pages.
| Situation | Recommended action | Reason |
|---|---|---|
| Site-wide 5xx or DNS failure | Treat as an incident, fix immediately | Blocks all crawling |
| Moved page returns 404 | Add a 301 redirect to the new URL | Preserves links and traffic |
| Deleted page with no equivalent | Return 404 or 410 | Redirecting to the homepage creates soft 404s |
| Internal links to dead URLs | Update links at the source | Saves crawl budget and improves navigation |
| Crawler blocked by bot protection | Allow verified crawler IP ranges | Avoids hidden site-wide errors |
The most frequent mistake is redirecting every 404 to the homepage, which search engines treat as soft 404s. Another is serving 200 status codes on error pages, which hides real problems from both crawlers and monitoring.
In everyday language
Search engines send robots to visit web pages and read them. A crawl error is a failed visit. The robot knocked, but the door was locked, the address was wrong or nobody answered.
One missing page is usually harmless, just like a single closed shop on a busy street. If the whole building is shut, though, the robot cannot read anything. It comes back less often, and pages may disappear from search results. Fixing crawl errors keeps the doors open for visitors and robots alike.
Seen in the real world
An online furniture retailer moves to a new shop platform over a weekend. A week later its search traffic is falling, and the crawl stats report shows a sharp rise in 5xx errors and 404 responses from Googlebot.
Log analysis reveals two causes. The new platform’s bot protection rejects many crawler requests with server errors, and about 1,500 old product URLs were never redirected. The team allows verified crawler IP ranges, adds 301 redirects from each old URL to its new equivalent, and fixes internal links in the menu. Within about three weeks crawling returns to normal and most product pages regain their positions.