Vector index

On this page 8

A vector index is a data structure that organizes embedding vectors so that a system can return the stored items most similar to a query vector without comparing the query against every item. In artificial intelligence, and especially in the retrieval & knowledge layer behind semantic search and RAG, the vector index is the component that turns millions of embeddings into a searchable collection with millisecond-level latency.

Most production vector indexes perform approximate nearest neighbor search, which deliberately accepts a small loss of accuracy in exchange for large gains in speed and memory efficiency. Choosing and tuning the index therefore means balancing recall, latency, memory and build cost against the size and update pattern of the data.

A vector index works by restricting the candidate set that a query must examine. A brute-force scan, also called exact nearest neighbor search, computes a distance between the query and every stored vector, which costs time proportional to the number of vectors multiplied by their dimension. For ten million 1,024-dimensional embeddings, that is roughly ten billion multiply-add operations per query, which is too slow for interactive use on ordinary hardware.

Index structures avoid this cost in three main ways. Graph-based indexes connect each vector to a few close neighbors and walk the graph toward the query. Partition-based indexes cluster the space and search only the clusters nearest to the query. Compression-based indexes shrink each vector into a compact code so that many more comparisons fit in memory and cache. Real systems often combine partitioning with compression, as in an IVF index with product quantization.

Every index is built for a specific distance metric: cosine similarity, inner product or Euclidean distance. The metric must match the one the embedding model was trained for; otherwise the ranking will be systematically wrong even when the index itself works perfectly.

Main families of vector index

The common index families differ mainly in how they trade memory for speed and how well they tolerate updates. The table below summarizes the families that practitioners encounter most often.

Index familyCore ideaStrengthMain weakness
Flat indexStores raw vectors and scans all of themExact results, no build timeLinear query cost
HNSWMulti-layer proximity graph with greedy searchHigh recall at low latencyHigh memory use, slow deletions
IVF indexClusters vectors and probes nearby clustersCompact and fast to buildRecall drops if clusters are probed too sparingly
IVF with product quantizationClusters plus compressed vector codesVery large collections in RAMLower precision without reranking
DiskANNGraph index stored mainly on SSDBillions of vectors on one machineDepends on fast storage and careful tuning
Binary or scalar quantized indexVectors reduced to bits or 8-bit integersLarge memory savingsCoarser distance estimates
Common vector index families and their properties

No single family is best everywhere. HNSW is the default in many vector databases because it delivers strong recall with little tuning, while quantized and disk-based indexes become necessary once the raw vectors no longer fit in memory.

Parameters that control recall and speed

Each index exposes a small set of build-time and query-time parameters that move it along the recall and latency curve. Build-time parameters fix the structure and require a rebuild to change, while query-time parameters can be adjusted per request.

ParameterIndex typeTypical rangeEffect of a higher value
M (graph degree)HNSW8 to 64Better recall, more memory
efConstructionHNSW100 to 500Better graph quality, slower build
efSearchHNSW32 to 512Better recall, slower queries
nlist (cluster count)IVFAbout the square root of the vector countSmaller clusters, more training cost
nprobeIVF1 to several percent of nlistBetter recall, slower queries
Code sizeProduct quantization8 to 64 bytes per vectorMore precision, more memory
Typical tuning parameters and their effects

The standard quality measure is recall at k, the share of the true k nearest neighbors that the index actually returns. Well-tuned indexes commonly reach 0.95 to 0.99 recall at a fraction of brute-force cost, but the right target depends on the application. A RAG pipeline that reranks the top 50 candidates can tolerate lower first-stage recall than a deduplication system that must not miss a near-duplicate.

A vector index is one component of a retrieval system, not the whole system, and it is often confused with the terms around it. The comparison below separates its role from neighboring concepts in the same topic.

ConceptWhat it isRelationship to the vector index
Embedding modelNetwork that converts text or images into vectorsProduces the vectors the index stores
Vector databaseManaged system with storage, filtering and replicationContains one or more vector indexes
Inverted indexTerm-to-document map used by BM25Sparse counterpart, often combined in hybrid search
RerankingSecond-stage scoring with a stronger modelRefines the candidates the index returns
Metadata filteringRestriction by attributes such as date or tenantApplied before, during or after the index search
How a vector index relates to neighboring concepts

The distinction matters in practice. Replacing a vector database rarely changes retrieval quality if the index type, metric and parameters stay the same, whereas changing the embedding model always requires re-embedding and rebuilding the index.

Choosing and operating a vector index

The choice of index follows mostly from collection size, memory budget and update rate. The decision table below reflects common engineering practice rather than fixed rules.

SituationRecommended indexReason
Under about 100,000 vectorsFlat indexExact results are fast enough
Millions of vectors, ample RAMHNSWBest recall-to-latency balance
Hundreds of millions, limited RAMIVF with product quantizationCompression keeps the index in memory
Billions of vectors on one serverDiskANN-style indexSSD holds the graph and full vectors
Frequent deletes and insertsHNSW or IVF with periodic rebuildsPrevents degraded graphs and stale clusters
Index choice by situation

Operating an index also involves lifecycle work. Deleted items are often only marked as removed, so graphs and clusters degrade over time and need scheduled compaction or rebuilds. IVF centroids trained on old data may no longer fit new content, which silently lowers recall.

Limitations and common mistakes

Most retrieval failures blamed on the vector index actually come from configuration and data problems around it. The most frequent ones are listed below.

  • Metric mismatch: indexing with Euclidean distance when the embedding model expects cosine similarity, or forgetting to normalize vectors.
  • Restrictive filters after search: post-filtering a top-10 result can leave few or no matches, so filtered search needs a larger candidate pool or filter-aware indexing.
  • Untested recall: tuning only for latency without measuring recall against an exact baseline on a held-out query set.
  • Mixed embedding versions: storing vectors from two model versions in one index, which makes distances meaningless.
  • Expecting relevance from similarity: a perfect index still returns the nearest vectors, not necessarily the most useful documents, which is why reranking and hybrid search exist.

In plain words

A vector index is like the catalog of a huge library organized by meaning instead of by title. Every document has been turned into a list of numbers that describes what it is about. When a question arrives, it is turned into numbers too, and the index quickly points to the documents whose numbers are closest.

Without the index, the computer would have to compare the question with every single document, one by one. The index uses shortcuts, such as grouping similar items or linking neighbors, so it checks only a small part of the collection. The key trade-off is that these shortcuts may occasionally miss a good match, in return for answers that arrive in milliseconds.

A real-world example

An insurance company builds an internal assistant that answers employee questions from about 40 million chunks of policy documents, each embedded as a 768-dimensional vector. Stored as raw 32-bit floats, the vectors need about 120 GB of memory, which exceeds the budget for the search servers.

The engineering team chooses an IVF index with product quantization, using 16,384 clusters and 64-byte codes, which shrinks the index to roughly 3 GB. Queries probe 64 clusters and retrieve 200 candidates, which a cross-encoder then reranks to the final top 5. Measured against exact search on 2,000 test questions, first-stage recall at 200 reaches about 0.96, and median retrieval latency stays under 30 milliseconds. The result is an assistant that answers from the right documents while running on a fraction of the hardware a brute-force design would require.