A vector index is a data structure that organizes embedding vectors so that a system can return the stored items most similar to a query vector without comparing the query against every item. In artificial intelligence, and especially in the retrieval & knowledge layer behind semantic search and RAG, the vector index is the component that turns millions of embeddings into a searchable collection with millisecond-level latency.
Most production vector indexes perform approximate nearest neighbor search, which deliberately accepts a small loss of accuracy in exchange for large gains in speed and memory efficiency. Choosing and tuning the index therefore means balancing recall, latency, memory and build cost against the size and update pattern of the data.
How a vector index narrows the search
A vector index works by restricting the candidate set that a query must examine. A brute-force scan, also called exact nearest neighbor search, computes a distance between the query and every stored vector, which costs time proportional to the number of vectors multiplied by their dimension. For ten million 1,024-dimensional embeddings, that is roughly ten billion multiply-add operations per query, which is too slow for interactive use on ordinary hardware.
Index structures avoid this cost in three main ways. Graph-based indexes connect each vector to a few close neighbors and walk the graph toward the query. Partition-based indexes cluster the space and search only the clusters nearest to the query. Compression-based indexes shrink each vector into a compact code so that many more comparisons fit in memory and cache. Real systems often combine partitioning with compression, as in an IVF index with product quantization.
Every index is built for a specific distance metric: cosine similarity, inner product or Euclidean distance. The metric must match the one the embedding model was trained for; otherwise the ranking will be systematically wrong even when the index itself works perfectly.
Main families of vector index
The common index families differ mainly in how they trade memory for speed and how well they tolerate updates. The table below summarizes the families that practitioners encounter most often.
| Index family | Core idea | Strength | Main weakness |
|---|---|---|---|
| Flat index | Stores raw vectors and scans all of them | Exact results, no build time | Linear query cost |
| HNSW | Multi-layer proximity graph with greedy search | High recall at low latency | High memory use, slow deletions |
| IVF index | Clusters vectors and probes nearby clusters | Compact and fast to build | Recall drops if clusters are probed too sparingly |
| IVF with product quantization | Clusters plus compressed vector codes | Very large collections in RAM | Lower precision without reranking |
| DiskANN | Graph index stored mainly on SSD | Billions of vectors on one machine | Depends on fast storage and careful tuning |
| Binary or scalar quantized index | Vectors reduced to bits or 8-bit integers | Large memory savings | Coarser distance estimates |
No single family is best everywhere. HNSW is the default in many vector databases because it delivers strong recall with little tuning, while quantized and disk-based indexes become necessary once the raw vectors no longer fit in memory.
Parameters that control recall and speed
Each index exposes a small set of build-time and query-time parameters that move it along the recall and latency curve. Build-time parameters fix the structure and require a rebuild to change, while query-time parameters can be adjusted per request.
| Parameter | Index type | Typical range | Effect of a higher value |
|---|---|---|---|
| M (graph degree) | HNSW | 8 to 64 | Better recall, more memory |
| efConstruction | HNSW | 100 to 500 | Better graph quality, slower build |
| efSearch | HNSW | 32 to 512 | Better recall, slower queries |
| nlist (cluster count) | IVF | About the square root of the vector count | Smaller clusters, more training cost |
| nprobe | IVF | 1 to several percent of nlist | Better recall, slower queries |
| Code size | Product quantization | 8 to 64 bytes per vector | More precision, more memory |
The standard quality measure is recall at k, the share of the true k nearest neighbors that the index actually returns. Well-tuned indexes commonly reach 0.95 to 0.99 recall at a fraction of brute-force cost, but the right target depends on the application. A RAG pipeline that reranks the top 50 candidates can tolerate lower first-stage recall than a deduplication system that must not miss a near-duplicate.
Vector index compared with related retrieval components
A vector index is one component of a retrieval system, not the whole system, and it is often confused with the terms around it. The comparison below separates its role from neighboring concepts in the same topic.
| Concept | What it is | Relationship to the vector index |
|---|---|---|
| Embedding model | Network that converts text or images into vectors | Produces the vectors the index stores |
| Vector database | Managed system with storage, filtering and replication | Contains one or more vector indexes |
| Inverted index | Term-to-document map used by BM25 | Sparse counterpart, often combined in hybrid search |
| Reranking | Second-stage scoring with a stronger model | Refines the candidates the index returns |
| Metadata filtering | Restriction by attributes such as date or tenant | Applied before, during or after the index search |
The distinction matters in practice. Replacing a vector database rarely changes retrieval quality if the index type, metric and parameters stay the same, whereas changing the embedding model always requires re-embedding and rebuilding the index.
Choosing and operating a vector index
The choice of index follows mostly from collection size, memory budget and update rate. The decision table below reflects common engineering practice rather than fixed rules.
| Situation | Recommended index | Reason |
|---|---|---|
| Under about 100,000 vectors | Flat index | Exact results are fast enough |
| Millions of vectors, ample RAM | HNSW | Best recall-to-latency balance |
| Hundreds of millions, limited RAM | IVF with product quantization | Compression keeps the index in memory |
| Billions of vectors on one server | DiskANN-style index | SSD holds the graph and full vectors |
| Frequent deletes and inserts | HNSW or IVF with periodic rebuilds | Prevents degraded graphs and stale clusters |
Operating an index also involves lifecycle work. Deleted items are often only marked as removed, so graphs and clusters degrade over time and need scheduled compaction or rebuilds. IVF centroids trained on old data may no longer fit new content, which silently lowers recall.
Limitations and common mistakes
Most retrieval failures blamed on the vector index actually come from configuration and data problems around it. The most frequent ones are listed below.
- Metric mismatch: indexing with Euclidean distance when the embedding model expects cosine similarity, or forgetting to normalize vectors.
- Restrictive filters after search: post-filtering a top-10 result can leave few or no matches, so filtered search needs a larger candidate pool or filter-aware indexing.
- Untested recall: tuning only for latency without measuring recall against an exact baseline on a held-out query set.
- Mixed embedding versions: storing vectors from two model versions in one index, which makes distances meaningless.
- Expecting relevance from similarity: a perfect index still returns the nearest vectors, not necessarily the most useful documents, which is why reranking and hybrid search exist.
In plain words
A vector index is like the catalog of a huge library organized by meaning instead of by title. Every document has been turned into a list of numbers that describes what it is about. When a question arrives, it is turned into numbers too, and the index quickly points to the documents whose numbers are closest.
Without the index, the computer would have to compare the question with every single document, one by one. The index uses shortcuts, such as grouping similar items or linking neighbors, so it checks only a small part of the collection. The key trade-off is that these shortcuts may occasionally miss a good match, in return for answers that arrive in milliseconds.
A real-world example
An insurance company builds an internal assistant that answers employee questions from about 40 million chunks of policy documents, each embedded as a 768-dimensional vector. Stored as raw 32-bit floats, the vectors need about 120 GB of memory, which exceeds the budget for the search servers.
The engineering team chooses an IVF index with product quantization, using 16,384 clusters and 64-byte codes, which shrinks the index to roughly 3 GB. Queries probe 64 clusters and retrieve 200 candidates, which a cross-encoder then reranks to the final top 5. Measured against exact search on 2,000 test questions, first-stage recall at 200 reaches about 0.96, and median retrieval latency stays under 30 milliseconds. The result is an assistant that answers from the right documents while running on a fraction of the hardware a brute-force design would require.