Sparse retrieval

On this page 7

Sparse retrieval is a family of search methods that represent queries and documents as high-dimensional vectors with mostly zero values, one dimension per vocabulary term, and rank documents by the weighted overlap of shared terms. In the field of Retrieval & knowledge within artificial intelligence, it is the classic lexical approach behind BM25 and TF-IDF and the modern learned approach behind models such as SPLADE.

Because only the terms that actually occur receive non-zero weights, sparse vectors can be stored in an inverted index, the same data structure that has powered web and enterprise search for decades. This makes sparse retrieval fast, cheap and interpretable, and it remains a core component of retrieval-augmented generation (RAG) pipelines, usually alongside dense retrieval.

How sparse retrieval scores documents

Sparse retrieval scores a document by summing the weights of query terms that also appear in that document. Each document becomes a vector over a vocabulary of tens of thousands to millions of terms, yet a typical passage activates only a few dozen to a few hundred dimensions, which is why the representation is called sparse.

At indexing time, the system tokenizes each document, assigns a weight to every term, and appends the document identifier to a posting list for that term. At query time, it reads only the posting lists of the query terms and accumulates scores. Algorithms such as WAND and Block-Max WAND skip documents that cannot enter the top results, so latency stays in the low milliseconds even on collections with hundreds of millions of documents.

The table below summarizes the stages of a sparse retrieval pipeline and what each one controls.

StageWhat happensWhat it affects
TokenizationText is split into words or subword unitsVocabulary coverage and handling of rare terms
NormalizationLowercasing, stemming or lemmatizationWhether word variants match each other
Term weightingEach term receives a statistical or learned weightRanking quality and relevance signals
IndexingPostings are written to an inverted indexIndex size and update speed
Query processingPosting lists are traversed with pruningLatency and throughput
Stages of a sparse retrieval pipeline

Statistical and learned sparse models

Sparse retrieval models fall into two broad groups: statistical models that derive weights from term counts, and learned sparse models that use a neural network to predict term weights. Both produce the same kind of index, which is why learned models can run on mature search engines without new infrastructure.

Statistical models such as TF-IDF and BM25 rely on term frequency, inverse document frequency and document length normalization. BM25 adds saturation through the k1 parameter, typically between 1.2 and 2.0, and length normalization through b, usually around 0.75. Learned models such as SPLADE, DeepImpact and uniCOIL use a transformer to assign weights and, in some cases, to expand documents with related terms that never appear in the original text, which reduces the vocabulary mismatch problem.

ModelWeight sourceTerm expansionTypical use
TF-IDFTerm and document frequenciesNoneBaselines, small collections, feature extraction
BM25Frequencies with saturation and length normalizationNoneDefault lexical ranking in search engines
DeepImpactNeural impact scores per termThrough document expansion beforehandResearch and efficient learned ranking
SPLADETransformer logits over the vocabularyLearned expansion of queries and documentsHigh-quality first-stage retrieval
Main sparse retrieval models compared

Learned sparse models usually require a GPU for document encoding at indexing time, while query encoding can be lightweight or even skipped in document-only variants. They often match or exceed dense retrievers on out-of-domain benchmarks such as BEIR, but they produce larger indexes than BM25 because expansion adds terms.

Sparse retrieval compared with dense retrieval

Sparse retrieval matches explicit vocabulary terms, while dense retrieval matches learned semantic embeddings with a few hundred to a few thousand dimensions, all of them non-zero. The two approaches fail in different ways, which is the main reason hybrid search combines them.

Sparse methods excel at exact identifiers, product codes, rare names and technical jargon, where an embedding model may blur distinct strings into similar vectors. Dense methods excel at paraphrases and conceptual queries, where the user and the document describe the same idea with different words. A common production pattern runs both retrievers and merges their lists with reciprocal rank fusion before a cross-encoder reranker.

PropertySparse retrievalDense retrieval
Vector shapeVocabulary-sized, mostly zerosFixed size, all values non-zero
Index structureInverted indexVector index such as HNSW or IVF
Exact term matchingStrongOften weak for rare strings
Synonyms and paraphrasesWeak unless expansion is learnedStrong
InterpretabilityHigh, each weight maps to a termLow, dimensions have no direct meaning
Training data neededNone for BM25Large labeled or synthetic pairs
Sparse and dense retrieval side by side

When to choose sparse retrieval

Sparse retrieval is the right first choice when queries contain precise terms that must match exactly or when no training data exists for the domain. BM25 needs no model training, adapts instantly to new documents and gives a strong baseline against which any dense or hybrid system should be measured.

SituationRecommended approachReason
Legal, medical or code search with exact identifiersBM25 or learned sparse as primaryLiteral matches carry most of the relevance
New domain without labeled dataBM25 baseline firstWorks without training and is hard to beat out of domain
Conversational or paraphrased questionsHybrid with dense retrievalSparse alone misses synonyms
Strict latency or cost budgetBM25 on an inverted indexRuns on CPUs with millisecond latency
Highest first-stage quality in RAGSPLADE plus dense, fused and rerankedCombines lexical precision and semantic recall
Decision guide for sparse retrieval

In RAG systems, retrieval quality depends heavily on chunking and metadata filtering. Short, coherent chunks give term statistics a clearer signal, and filters on fields such as date or source reduce the candidate set before scoring.

Limitations and common mistakes

The central weakness of sparse retrieval is vocabulary mismatch: a relevant document that uses different words from the query receives a score of zero. Learned expansion and query rewriting reduce this gap, but they do not remove it entirely.

  • Wrong language analyzers break matching, for example applying an English stemmer to Czech or German text.
  • Untuned BM25 parameters can hurt collections with very long or very short documents; tuning k1 and b on a validation set often helps.
  • Ignoring stop words in technical queries removes meaningful tokens such as the programming language name “Go”.
  • Assuming learned sparse indexes are small leads to capacity surprises, because expansion can multiply postings several times.
  • Skipping evaluation with metrics such as recall at k and mean reciprocal rank hides whether sparse, dense or hybrid retrieval actually performs best.

The idea in brief

Sparse retrieval works like the index at the back of a book. Each important word points to the pages where it appears. When a question arrives, the system looks up its words and returns the pages that contain them most strongly.

Newer versions use AI to decide which words matter and to add helpful related words. The key point is simple: sparse retrieval finds documents by shared words, which makes it fast, reliable for exact names and codes, and easy to understand, but weaker when people describe the same thing in different words.

A practical scenario

A software company builds an internal assistant that answers support engineers’ questions from about two million tickets and manuals. Its first version uses only dense embeddings, and engineers soon complain that queries containing error codes such as “ERR_4012” return loosely related tickets about other errors.

The team adds a BM25 index next to the vector index, merges both result lists with reciprocal rank fusion and passes the top 50 candidates to a cross-encoder reranker. On a test set of 400 real questions, recall at 20 rises from about 0.71 to 0.86, with the largest gains on queries containing codes and product versions. Query latency grows by only a few milliseconds, because the inverted index runs on the existing CPU servers.