Information retrieval (IR) is the discipline of finding and ranking documents that satisfy an information need, expressed as a query, within a large and mostly unstructured collection. In artificial intelligence, and in the retrieval & knowledge stack in particular, IR supplies the evidence layer that search engines, question answering systems and retrieval-augmented generation depend on.
Unlike a database lookup, which returns rows that exactly match a condition, IR deals with graded relevance: every document receives a score, and the system returns a ranked list. The quality of that ranking, not the mere presence of a match, decides whether the user or a downstream language model gets useful material.
How an information retrieval system works
An information retrieval system works in two phases: indexing and querying. During indexing, documents are split, normalized and converted into a searchable structure; during querying, the user request is converted into the same representation and matched against that structure to produce scored candidates.
Classical systems build an inverted index that maps each term to the documents containing it, which makes lookup fast even across billions of pages. Modern neural systems add a vector index that stores dense embeddings and answers queries through approximate nearest neighbor search. Most production pipelines combine both and finish with a reranking stage that applies a slower, more accurate model to the top candidates.
The table below summarizes the main stages of a typical retrieval pipeline and what each one contributes.
| Stage | What happens | Typical technique |
|---|---|---|
| Ingestion | Documents are cleaned, split and enriched with metadata | Parsing, chunking, deduplication |
| Indexing | Text is turned into searchable structures | Inverted index, vector index |
| Query processing | The request is normalized or reformulated | Tokenization, query rewriting, expansion |
| Candidate retrieval | A fast method selects hundreds of candidates | BM25, dense retrieval, hybrid search |
| Reranking | A precise model reorders the top results | Cross-encoder, learned ranking model |
| Presentation | Results are shown or passed to a model | Snippets, citations, context window |
Retrieval models: from Boolean to neural
A retrieval model defines how relevance is computed, and the field has moved through several generations of models. Each generation is still in use, because older models remain cheap, transparent and robust while newer ones capture meaning that keyword matching misses.
The Boolean model returns documents that satisfy a logical expression without ranking them. The vector space model with TF-IDF weighting introduced graded scores. Probabilistic models, above all BM25, estimate the likelihood that a document is relevant and remain a strong baseline. Neural models encode queries and documents as embeddings, either as single vectors (bi-encoders) or as many token vectors (late interaction, as in ColBERT), or as learned sparse weights (as in SPLADE).
The following comparison shows where each family of models fits.
| Model family | Relevance signal | Strength | Weakness |
|---|---|---|---|
| Boolean | Exact logical match | Predictable, auditable | No ranking, brittle queries |
| TF-IDF vector space | Weighted term overlap | Simple and fast | Ignores synonyms |
| BM25 | Term frequency with saturation and length normalization | Strong zero-shot baseline | Vocabulary mismatch |
| Dense retrieval | Embedding similarity | Captures paraphrase and meaning | Weak on rare names and codes |
| Learned sparse | Expanded term weights | Index-compatible semantics | Larger index, training cost |
| Late interaction | Token-level similarity | High precision | High storage cost |
Measuring retrieval quality
Retrieval quality is measured by comparing system rankings against relevance judgments for a set of test queries. The two founding measures are precision, the share of returned documents that are relevant, and recall, the share of relevant documents that were returned; they usually trade against each other.
Ranked evaluation adds position sensitivity, because a relevant document at rank 1 is worth far more than the same document at rank 50. Standard benchmark collections, such as those from the TREC evaluation campaigns, MS MARCO and BEIR, allow systems to be compared on shared queries and judgments.
| Metric | What it measures | When to use it |
|---|---|---|
| Precision at k | Share of relevant results in the top k | Short result lists, user-facing search |
| Recall at k | Share of all relevant items found in the top k | Candidate stages before reranking, RAG |
| Mean reciprocal rank | Position of the first relevant result | Single-answer tasks, question answering |
| nDCG | Graded relevance discounted by rank | Multi-level judgments, web search |
| Mean average precision | Precision averaged over all relevant positions | Binary judgments, research benchmarks |
Offline metrics should be complemented by online signals such as click-through and task completion, because a ranking that scores well on judgments can still fail real users whose intent differs from the test set.
Information retrieval in RAG and AI systems
In retrieval-augmented generation, information retrieval is the component that decides what the language model knows for a given answer. The generator can only ground its output in passages the retriever returns, so retrieval errors propagate directly into hallucinations or incomplete answers.
This changes the priorities compared with classic web search. A RAG retriever typically optimizes recall at small k (often 5 to 20 passages) because the context window is limited, and it must return self-contained chunks rather than whole documents. Techniques such as hybrid search, reciprocal rank fusion, query rewriting and metadata filtering exist largely to raise the chance that the right passage lands in that small window.
The decision table below maps common requirements to a sensible retrieval approach.
| Situation | Recommended approach | Reason |
|---|---|---|
| Product codes, legal citations, names | BM25 or hybrid search | Exact tokens matter more than meaning |
| Paraphrased natural-language questions | Dense retrieval with reranking | Bridges vocabulary differences |
| Mixed corpus with unknown query types | Hybrid search with rank fusion | Covers both failure modes |
| Strict access control or recency needs | Metadata filtering before ranking | Removes ineligible documents early |
| Questions spanning several documents | Multi-hop or iterative retrieval | One query rarely finds all evidence |
Limitations and common mistakes
Most retrieval failures come from mismatches between the index and the questions, not from the ranking algorithm itself. Teams often tune embedding models while the real problem lies in poor chunking, missing metadata or stale documents.
- Vocabulary mismatch: users and authors describe the same idea with different words, which hurts lexical models.
- Semantic drift: dense models retrieve passages that are topically similar but do not answer the question.
- Evaluation without judgments: judging a retriever only by final answer quality hides whether retrieval or generation failed.
- Ignoring freshness and permissions: a highly relevant but outdated or unauthorized document is a defect, not a success.
- Domain shift: models trained on web data can underperform on medical, legal or internal corporate text without adaptation.
A disciplined practice is to evaluate retrieval separately, with a small labeled query set from the real domain, before adjusting prompts or generators.
In simple terms
Information retrieval is the craft of finding the right documents in a huge pile and putting the best ones first. A library catalog, a web search engine and a chatbot that looks up company policies all rely on it.
Think of a skilled librarian. You describe what you need in your own words, and the librarian does not hand you every book that mentions the topic. Instead, the librarian picks the few that actually answer your question and puts the most useful one on top. Good retrieval means the right answer appears near the top, not somewhere on page ten.
A practical example
A mid-sized insurance company builds an internal assistant that answers employee questions about roughly 40,000 policy documents. The first version uses dense retrieval alone, and staff soon notice that questions mentioning specific clause numbers, such as a reference to section 4.2.7, return loosely related passages instead of the exact clause.
The team labels 300 real employee questions with the passages that answer them and measures recall at 10 of about 62 percent. They add BM25 alongside the embeddings, merge the two lists with reciprocal rank fusion, filter by policy version date and rerank the top 50 candidates with a cross-encoder.
On the same test set, recall at 10 rises to about 88 percent, and complaints about wrong clauses drop sharply. The generator was never changed: the improvement came entirely from better information retrieval.