Googlebot is the web crawler used by Google Search to discover URLs, fetch their content and pass it on for rendering and indexing. In SEO and crawling, it is the most consequential crawler on the web, because a page that Googlebot cannot fetch or render cannot appear in Google’s organic results on its merits.
The name covers two main crawler types: Googlebot Smartphone, which crawls the vast majority of pages under mobile-first indexing, and Googlebot Desktop, which is used far less. Both identify themselves with the same robots.txt product token, Googlebot, so a single set of rules controls them.
How Googlebot crawls and renders a page
Googlebot works in a pipeline of discovery, crawling, rendering and indexing. URLs enter a crawl queue from links on known pages, XML sitemaps and other sources. Before fetching a URL, Googlebot checks the site’s robots.txt file and skips any path it is disallowed from crawling.
The fetched HTML is parsed for links and content. Pages that rely on JavaScript are then processed by the Web Rendering Service, which runs an evergreen version of Chromium to execute scripts and produce the rendered DOM. Rendering may occur shortly after the fetch or later, depending on resources, so content that appears only after rendering can be indexed with a delay.
The stages and the most common blockers are summarized below.
| Stage | What happens | Common blocker |
|---|---|---|
| Discovery | New URLs found through links and sitemaps | Orphan pages without internal links |
| Robots.txt check | Crawl permission evaluated per path | Overly broad disallow rules |
| Fetch | HTML and resources downloaded over HTTP | Server errors, timeouts, blocked resources |
| Rendering | JavaScript executed in headless Chromium | Content requiring user interaction |
| Hand-off to indexing | Content, links and signals passed on | Noindex directive or duplicate content |
Googlebot variants and related Google crawlers
Google operates several crawlers with separate user agent tokens, and robots.txt rules can address each one. Knowing which crawler a request belongs to prevents accidental blocking of the wrong service.
| Crawler or token | Purpose | Robots.txt token |
|---|---|---|
| Googlebot Smartphone | Primary crawler for web search | Googlebot |
| Googlebot Desktop | Secondary crawling of desktop pages | Googlebot |
| Googlebot Image | Images for image search | Googlebot-Image |
| Googlebot Video | Video files and pages | Googlebot-Video |
| Google-Extended | Control token for use in generative AI models | Google-Extended |
Google-Extended is a control token, not a separate crawler: blocking it does not stop Googlebot from crawling for Search, and it has no effect on ranking. Other special-purpose fetchers, such as the inspection tool used for live URL tests, follow their own rules and are documented separately.
Crawl rate, crawl budget and server signals
Googlebot adjusts how fast it crawls a site based on crawl capacity and crawl demand. Capacity reflects how quickly and reliably the server responds; demand reflects how popular and how frequently updated the URLs are. Together they determine the site’s effective crawl budget.
The server can influence crawling through its responses. The table shows how Googlebot typically reacts to common HTTP status codes.
| Response | Googlebot reaction | SEO consequence |
|---|---|---|
| 200 OK | Content processed for indexing | Page eligible for search |
| 301 or 308 redirect | Target URL followed and treated as preferred | Signals consolidate on the target |
| 404 or 410 | URL eventually dropped from the index | Lost visibility for that URL |
| 429 or 503 | Crawl rate reduced, retried later | Temporary slowdown if brief |
| 5xx errors persisting | Crawling slows sharply | Pages may drop from the index |
Googlebot ignores the crawl-delay directive in robots.txt. To slow it down temporarily, a site should return 503 or 429 responses for a short period rather than blocking it.
Googlebot compared with other web crawlers
Googlebot is one web crawler among many, and its behavior differs from other bots in ways that matter for site owners.
| Crawler type | Renders JavaScript | Respects robots.txt | Main purpose |
|---|---|---|---|
| Googlebot | Yes, with evergreen Chromium | Yes | Indexing for Google Search |
| Other search engine crawlers | Varies by engine | Generally yes | Indexing for their own engines |
| AI training crawlers | Often limited | Most major ones do | Collecting text for model training |
| SEO audit crawlers | Optional | Configurable | Site analysis by owners |
| Scrapers spoofing Googlebot | Varies | Often not | Copying content or probing sites |
Two differences have practical weight. Googlebot renders pages much like a modern browser, so JavaScript-dependent content is usually visible to it, while some other crawlers see only the raw HTML. And because Googlebot drives the largest share of organic traffic for most sites, blocking or slowing it has outsized consequences, whereas blocking a minor bot rarely affects visibility. Rules written for one crawler should therefore be reviewed for their effect on the others.
Verifying Googlebot and avoiding mistakes
The user agent string can be faked, so genuine Googlebot traffic should be verified. The standard method is a reverse DNS lookup of the requesting IP, which should resolve to a host under googlebot.com, google.com or googleusercontent.com, followed by a forward DNS lookup confirming the same IP. Google also publishes its crawler IP ranges in machine-readable form.
- Blocking CSS and JavaScript in robots.txt, which prevents correct rendering.
- Using robots.txt to remove pages from the index, when a noindex directive is required.
- Serving different content to Googlebot than to users, which risks being treated as cloaking.
- Firewall rules that block Googlebot IPs after a spike in traffic.
- Relying on client-side rendering for critical content and links without testing.
The most reliable diagnostic is the URL inspection feature in Search Console, which shows the fetched and rendered version, and the crawl stats report, which reveals response times and error trends.
Explained without jargon
Googlebot is the program Google sends to visit websites. It reads pages, follows the links on them and reports back what it finds. Google then uses that information to decide which pages to show in search results.
If Googlebot cannot reach a page, or sees only an empty shell, that page cannot be found through Google search. Making a site easy for Googlebot to read is like keeping a shop’s doors open and its shelves labeled for a visiting inspector.
A typical use case
An online retailer launches a redesigned site built as a single-page JavaScript application. Within a month, organic traffic to category pages drops noticeably. The SEO team opens the URL inspection tool and sees that the rendered HTML lacks product links, because the products load only after the visitor scrolls.
The developers switch category pages to server-side rendering with standard anchor links and pagination. Server logs, verified by reverse DNS, then show Googlebot fetching product URLs it had previously ignored. Over the following weeks, the number of indexed product pages recovers, and the team adds a crawl test to its release checklist.