Information retrieval

Also known as: IR

On this page 7

Information retrieval (IR) is the discipline of finding and ranking documents that satisfy an information need, expressed as a query, within a large and mostly unstructured collection. In artificial intelligence, and in the retrieval & knowledge stack in particular, IR supplies the evidence layer that search engines, question answering systems and retrieval-augmented generation depend on.

Unlike a database lookup, which returns rows that exactly match a condition, IR deals with graded relevance: every document receives a score, and the system returns a ranked list. The quality of that ranking, not the mere presence of a match, decides whether the user or a downstream language model gets useful material.

How an information retrieval system works

An information retrieval system works in two phases: indexing and querying. During indexing, documents are split, normalized and converted into a searchable structure; during querying, the user request is converted into the same representation and matched against that structure to produce scored candidates.

Classical systems build an inverted index that maps each term to the documents containing it, which makes lookup fast even across billions of pages. Modern neural systems add a vector index that stores dense embeddings and answers queries through approximate nearest neighbor search. Most production pipelines combine both and finish with a reranking stage that applies a slower, more accurate model to the top candidates.

The table below summarizes the main stages of a typical retrieval pipeline and what each one contributes.

StageWhat happensTypical technique
IngestionDocuments are cleaned, split and enriched with metadataParsing, chunking, deduplication
IndexingText is turned into searchable structuresInverted index, vector index
Query processingThe request is normalized or reformulatedTokenization, query rewriting, expansion
Candidate retrievalA fast method selects hundreds of candidatesBM25, dense retrieval, hybrid search
RerankingA precise model reorders the top resultsCross-encoder, learned ranking model
PresentationResults are shown or passed to a modelSnippets, citations, context window
Stages of a typical retrieval pipeline

Retrieval models: from Boolean to neural

A retrieval model defines how relevance is computed, and the field has moved through several generations of models. Each generation is still in use, because older models remain cheap, transparent and robust while newer ones capture meaning that keyword matching misses.

The Boolean model returns documents that satisfy a logical expression without ranking them. The vector space model with TF-IDF weighting introduced graded scores. Probabilistic models, above all BM25, estimate the likelihood that a document is relevant and remain a strong baseline. Neural models encode queries and documents as embeddings, either as single vectors (bi-encoders) or as many token vectors (late interaction, as in ColBERT), or as learned sparse weights (as in SPLADE).

The following comparison shows where each family of models fits.

Model familyRelevance signalStrengthWeakness
BooleanExact logical matchPredictable, auditableNo ranking, brittle queries
TF-IDF vector spaceWeighted term overlapSimple and fastIgnores synonyms
BM25Term frequency with saturation and length normalizationStrong zero-shot baselineVocabulary mismatch
Dense retrievalEmbedding similarityCaptures paraphrase and meaningWeak on rare names and codes
Learned sparseExpanded term weightsIndex-compatible semanticsLarger index, training cost
Late interactionToken-level similarityHigh precisionHigh storage cost
Families of retrieval models compared

Measuring retrieval quality

Retrieval quality is measured by comparing system rankings against relevance judgments for a set of test queries. The two founding measures are precision, the share of returned documents that are relevant, and recall, the share of relevant documents that were returned; they usually trade against each other.

Ranked evaluation adds position sensitivity, because a relevant document at rank 1 is worth far more than the same document at rank 50. Standard benchmark collections, such as those from the TREC evaluation campaigns, MS MARCO and BEIR, allow systems to be compared on shared queries and judgments.

MetricWhat it measuresWhen to use it
Precision at kShare of relevant results in the top kShort result lists, user-facing search
Recall at kShare of all relevant items found in the top kCandidate stages before reranking, RAG
Mean reciprocal rankPosition of the first relevant resultSingle-answer tasks, question answering
nDCGGraded relevance discounted by rankMulti-level judgments, web search
Mean average precisionPrecision averaged over all relevant positionsBinary judgments, research benchmarks
Common evaluation metrics in information retrieval

Offline metrics should be complemented by online signals such as click-through and task completion, because a ranking that scores well on judgments can still fail real users whose intent differs from the test set.

Information retrieval in RAG and AI systems

In retrieval-augmented generation, information retrieval is the component that decides what the language model knows for a given answer. The generator can only ground its output in passages the retriever returns, so retrieval errors propagate directly into hallucinations or incomplete answers.

This changes the priorities compared with classic web search. A RAG retriever typically optimizes recall at small k (often 5 to 20 passages) because the context window is limited, and it must return self-contained chunks rather than whole documents. Techniques such as hybrid search, reciprocal rank fusion, query rewriting and metadata filtering exist largely to raise the chance that the right passage lands in that small window.

The decision table below maps common requirements to a sensible retrieval approach.

SituationRecommended approachReason
Product codes, legal citations, namesBM25 or hybrid searchExact tokens matter more than meaning
Paraphrased natural-language questionsDense retrieval with rerankingBridges vocabulary differences
Mixed corpus with unknown query typesHybrid search with rank fusionCovers both failure modes
Strict access control or recency needsMetadata filtering before rankingRemoves ineligible documents early
Questions spanning several documentsMulti-hop or iterative retrievalOne query rarely finds all evidence
Choosing a retrieval approach

Limitations and common mistakes

Most retrieval failures come from mismatches between the index and the questions, not from the ranking algorithm itself. Teams often tune embedding models while the real problem lies in poor chunking, missing metadata or stale documents.

  • Vocabulary mismatch: users and authors describe the same idea with different words, which hurts lexical models.
  • Semantic drift: dense models retrieve passages that are topically similar but do not answer the question.
  • Evaluation without judgments: judging a retriever only by final answer quality hides whether retrieval or generation failed.
  • Ignoring freshness and permissions: a highly relevant but outdated or unauthorized document is a defect, not a success.
  • Domain shift: models trained on web data can underperform on medical, legal or internal corporate text without adaptation.

A disciplined practice is to evaluate retrieval separately, with a small labeled query set from the real domain, before adjusting prompts or generators.

In simple terms

Information retrieval is the craft of finding the right documents in a huge pile and putting the best ones first. A library catalog, a web search engine and a chatbot that looks up company policies all rely on it.

Think of a skilled librarian. You describe what you need in your own words, and the librarian does not hand you every book that mentions the topic. Instead, the librarian picks the few that actually answer your question and puts the most useful one on top. Good retrieval means the right answer appears near the top, not somewhere on page ten.

A practical example

A mid-sized insurance company builds an internal assistant that answers employee questions about roughly 40,000 policy documents. The first version uses dense retrieval alone, and staff soon notice that questions mentioning specific clause numbers, such as a reference to section 4.2.7, return loosely related passages instead of the exact clause.

The team labels 300 real employee questions with the passages that answer them and measures recall at 10 of about 62 percent. They add BM25 alongside the embeddings, merge the two lists with reciprocal rank fusion, filter by policy version date and rerank the top 50 candidates with a cross-encoder.

On the same test set, recall at 10 rises to about 88 percent, and complaints about wrong clauses drop sharply. The generator was never changed: the improvement came entirely from better information retrieval.