Indexing is the process in which a search engine analyzes crawled pages, decides which ones to keep and stores them in its index, the database from which search results are retrieved. In SEO, indexing sits between crawling and ranking: a crawled page becomes eligible to appear in search results only after it has been indexed.
Indexing is selective, not automatic. Search engines skip pages they consider duplicate, low in value, blocked by directives or technically unreadable, so a site owner should treat indexing as a decision the search engine makes about each URL rather than a guaranteed step after publication.
How search engine indexing works
Search engine indexing works as a pipeline that turns raw fetched documents into searchable records. After a crawler downloads a page, the search engine renders it, extracts its content and signals, groups it with duplicates, selects a canonical version and finally writes that version into the index.
| Stage | What the search engine does | What the site owner controls |
|---|---|---|
| Rendering | Executes JavaScript to build the final page | Server-side rendering and crawlable resources |
| Content extraction | Reads text, headings, links, images and structured data | Semantic HTML and visible primary content |
| Signal analysis | Detects language, topic, entities and quality signals | Clear topic focus and original information |
| Duplicate clustering | Groups pages with identical or near-identical content | Avoiding parameter and session duplicates |
| Canonical selection | Chooses one URL to represent each cluster | Canonical tags, redirects and internal links |
| Storage | Adds the canonical version to the index | Only indirectly, through page quality |
Technically, the core of a search index is an inverted index: instead of storing a list of words for each page, it stores a list of pages for each term, together with positions and other attributes. This structure lets a search engine find matching documents among hundreds of billions of pages in a fraction of a second. Modern indexes also store vector representations of passages for semantic retrieval, which AI search features rely on heavily.
Indexing compared with crawling and ranking
Indexing is often confused with the stages around it, yet each stage answers a different question about a URL. Diagnosing visibility problems starts with identifying which of the three has failed.
| Stage | Question it answers | Typical failure symptom |
|---|---|---|
| Crawling | Can the search engine fetch the URL | URL never requested in server logs |
| Indexing | Will the search engine store the page | Crawled but not present in the index |
| Ranking | Where does the page appear for a query | Indexed but shown only on deep result pages |
The distinction matters because the fixes are completely different. A crawling problem calls for technical access fixes, an indexing problem usually calls for better content, consolidation or clearer canonical signals, and a ranking problem calls for relevance and authority work. Checking whether a URL is indexed, for example with the URL inspection feature in Google Search Console, is the fastest way to separate the three.
Indexing statuses and what they mean
Google reports the indexing state of URLs in the Page indexing report of Search Console, often still called index coverage. Each excluded status points to a specific cause and a specific remedy.
| Status | Meaning | Typical response |
|---|---|---|
| Discovered, currently not indexed | Known URL that has not been crawled yet | Strengthen internal links and server capacity |
| Crawled, currently not indexed | Fetched but judged not worth storing for now | Improve, merge or remove thin content |
| Duplicate without user-selected canonical | Part of a duplicate cluster without a declared canonical | Add canonical tags or consolidate URLs |
| Alternate page with proper canonical tag | Correctly points to another canonical URL | No action if intended |
| Excluded by noindex tag | Page carries a noindex directive | Remove the directive if the page should rank |
| Blocked by robots.txt | Crawler is not allowed to fetch the URL | Unblock if indexing is wanted |
| Soft 404 | Page looks empty or missing despite status 200 | Return a real 404 or add substantive content |
Not every excluded URL is a problem. On a healthy site, many exclusions are intentional, such as redirects, alternates and filtered views. The meaningful metric is whether the pages that should rank are indexed, not the raw total of indexed URLs.
Signals that control indexing
Site owners influence indexing through directives, which search engines obey, and hints, which they weigh against other evidence. Treating a hint as a command is a common source of unexpected results.
| Signal | Type | Effect on indexing |
|---|---|---|
| Meta robots noindex | Directive | Keeps the page out of the index once it is crawled |
| X-Robots-Tag header | Directive | Same effect for PDFs, images and other files |
| Rel canonical | Hint | Suggests the preferred URL in a duplicate cluster |
| 301 redirect | Strong signal | Moves indexing to the target URL |
| XML sitemap | Hint | Helps discovery and supports canonical choice |
| Internal links | Hint | Indicate importance and the preferred URL |
| Temporary removal request | Temporary block | Hides a URL from Google results for about six months |
Robots.txt is deliberately missing from the directives above, because it controls crawling, not indexing. A page blocked in robots.txt can still be indexed from links alone, and its noindex tag will never be seen. Requests to speed up indexing also have limits: Google offers its Indexing API officially only for job posting and livestream pages, while Bing, Yandex and several other engines accept change notifications through the IndexNow protocol.
Common indexing mistakes
Most indexing failures come from conflicting signals or low-value URL inventory, not from penalties. The mistakes below appear repeatedly in technical audits:
- Leaving a noindex directive from a staging environment on a live site after launch.
- Combining robots.txt blocking with noindex, so the directive is never read.
- Canonical tags pointing to redirected, broken or noindexed URLs, which search engines then ignore.
- Publishing large numbers of thin or templated pages, which lowers the share of URLs that get indexed.
- Rendering primary content only with client-side JavaScript that fails or times out.
- Repeatedly requesting indexing instead of fixing the quality or duplication issue behind an exclusion.
A plain-language summary
Think of a search engine as a huge library. Crawling is when a librarian brings a new book into the building. Indexing is when the librarian reads it, decides it is worth keeping and adds it to the catalog so visitors can find it.
Not every book makes it into the catalog. Copies of books already on the shelves, nearly empty books and books marked do not catalog are left out. If a page is not in the index, searchers cannot find it, no matter how good it is. That is why website owners check which of their pages are indexed and fix the reasons when important ones are missing.
An example from practice
A B2B software company submitted 3,400 blog URLs in its XML sitemap, but the Page indexing report showed only 2,150 of them indexed, about 63%. More than 1,100 URLs carried the status crawled, currently not indexed. An audit found 600 short posts covering overlapping topics and 300 tag archive pages with almost no unique text.
The content team merged the 600 overlapping posts into 140 comprehensive guides, redirecting each old URL with a 301 to the guide that replaced it. They added noindex to the 300 tag archives, removed both groups from the sitemap and linked the new guides from relevant product pages. The sitemap now listed 2,640 URLs: the original 3,400, minus the 600 merged posts and 300 tag archives, plus the 140 new guides.
Three months later, 2,430 of the 2,640 submitted URLs were indexed, a rate of about 92%. Organic traffic to the blog grew as well, mainly because the consolidated guides began ranking for queries that the fragmented posts had never reached.