Retriever

On this page 7

A retriever is the component of an information access pipeline that takes a query and returns a small ranked set of relevant passages from a much larger collection. In the field of retrieval & knowledge within artificial intelligence, the retriever is the first stage of retrieval-augmented generation, where it supplies a language model with the evidence it needs to answer accurately.

The retriever does not generate text and does not judge final correctness. Its job is recall under a latency budget: find the passages that matter, quickly, so that later stages such as a reranker or a generator can work with a manageable amount of context.

How a retriever finds relevant passages

A retriever works by turning both the query and the stored documents into comparable representations and then scoring how well each document matches. The collection is processed ahead of time during ingestion, so that at query time the system only has to encode the query and search an index, rather than read every document.

The process has two phases. In the offline indexing phase, documents are split into chunks, each chunk is converted into a sparse term vector, a dense embedding or both, and the results are stored in an index. In the online query phase, the query is encoded the same way, the index returns the top-k candidates, and optional filters remove items that violate metadata constraints such as date, language or access rights.

The following table summarizes the main steps and what each one controls.

StageWhat happensMain design choice
ChunkingDocuments are split into passagesChunk size and overlap
EncodingText becomes vectors or term weightsEmbedding model or term weighting
IndexingRepresentations are stored for fast lookupExact or approximate index
Query encodingThe user query is convertedSame model as the documents
Candidate searchThe index returns top-k matchesValue of k and similarity metric
FilteringResults are restricted by metadataPre-filtering or post-filtering
Stages of a typical retrieval pipeline

Sparse, dense and hybrid retrievers

Retrievers fall into three main families that differ in how they represent meaning. A sparse retriever scores documents by weighted term overlap, typically with BM25 or TF-IDF, and excels at exact matches on names, codes and rare words. A dense retriever uses a bi-encoder embedding model to place queries and passages in a shared vector space, so it can match paraphrases and synonyms that share no words.

Hybrid retrievers run both approaches and merge the ranked lists, often with reciprocal rank fusion. Learned sparse models such as SPLADE and late-interaction models such as ColBERT sit between the families, keeping term-level precision while adding learned semantics. The table compares the families on the properties that usually decide the choice.

Retriever typeRepresentationStrengthWeakness
Sparse (BM25)Inverted index of weighted termsExact terms, no training neededMisses synonyms and paraphrases
Dense (bi-encoder)One embedding per passageSemantic similarityWeak on rare identifiers
Learned sparse (SPLADE)Expanded term weightsInterpretable and semanticLarger index, model required
Late interaction (ColBERT)One vector per tokenFine-grained matchingHigh storage cost
HybridSparse and dense lists fusedRobust across query typesTwo indexes to maintain
Comparison of retriever families

Retriever versus reranker and generator

The retriever is distinct from the components that follow it, and confusing them leads to poor system design. The retriever must be fast enough to search millions of passages, so it compares precomputed representations. A reranker, usually a cross-encoder, reads the query and each candidate together and is far more accurate, but it is too slow to apply to the whole collection.

This division produces the common retrieve-then-rerank pattern: the retriever returns perhaps 50 to 200 candidates with high recall, and the reranker reorders them so that the best 3 to 10 reach the generator. The generator then grounds its answer in those passages.

ComponentPrimary goalTypical input sizeRelative cost per item
RetrieverHigh recallEntire collectionVery low
RerankerHigh precisionTens to hundreds of candidatesModerate
GeneratorFaithful answerA few passagesHigh
Roles in a retrieval-augmented system

Measuring retriever quality

Retriever quality is measured with ranking metrics computed against a set of queries with known relevant passages. The most important metric for a first-stage retriever is recall at k, because a passage that is not retrieved can never be used later, no matter how good the reranker or generator is.

Teams also track mean reciprocal rank and nDCG to see how high the relevant passages appear, and they monitor latency percentiles because retrieval sits on the critical path of every request. Public benchmarks give a rough orientation, but domain-specific evaluation sets are far more predictive, since retriever performance varies sharply between general web text and specialized material such as contracts or source code.

MetricWhat it measuresWhen it matters most
Recall at kShare of relevant passages found in top kFirst-stage retrieval
Mean reciprocal rankPosition of the first relevant resultSingle-answer questions
nDCG at kGraded relevance weighted by positionMultiple relevant passages
p95 latencySlowest typical response timeInteractive applications
Common retriever metrics

Common retriever failures and fixes

Most poor answers in retrieval-augmented systems trace back to the retriever rather than the language model. The failures below are frequent in practice, and each has a well-understood remedy.

  • Vocabulary mismatch: users and documents describe the same thing differently; query rewriting, query expansion or a dense model helps.
  • Identifier blindness: dense retrievers miss product codes and error numbers; adding a sparse component fixes most cases.
  • Lost context: chunks lack the surrounding document title or section; contextual retrieval or parent document retrieval restores it.
  • Redundant results: the top k passages repeat one another; maximal marginal relevance improves diversity.
  • Embedding drift: documents and queries are encoded with different model versions; re-index the whole collection after any model change.

A further warning concerns access control. A retriever that ignores document permissions can surface confidential passages to unauthorized users, so permission filtering belongs inside retrieval, not after generation.

A non-technical view

Think of a retriever as a very fast librarian. You ask a question, and before anyone writes an answer, the librarian runs to the shelves and brings back the few pages most likely to help. The librarian does not write the reply. It only chooses what to read.

If the librarian brings the wrong pages, even a brilliant writer will give a weak answer. That is why good retrieval comes first: an AI assistant can only be as accurate as the material it is handed.

A concrete example

An internal support assistant at a mid-sized software company answers employee questions from about 40,000 policy and product documents. The first version used only a dense retriever returning the top 5 chunks. Questions that mentioned exact error codes, such as a specific license failure number, often returned unrelated passages because the embedding model treated the codes as noise.

The team added a BM25 index, fused both lists with reciprocal rank fusion, raised the candidate count to 100 and inserted a cross-encoder reranker that kept the best 6. On a test set of 300 real employee questions, recall at 100 rose noticeably and the share of answers judged correct by support staff improved, while median retrieval latency stayed under 150 milliseconds. The lesson was that the retriever, not the language model, was the bottleneck.