Multi-hop retrieval is a retrieval strategy in which a system answers a question by running a sequence of dependent searches, where each step uses evidence found in the previous step to formulate the next query. In artificial intelligence, and particularly in the retrieval & knowledge layer of question answering and RAG systems, it addresses questions whose answer is spread across several documents that share no obvious vocabulary with the original question.
A typical example is the question “Which university did the founder of the company that built the first commercial jet airliner attend?” No single passage contains the answer. The system must first identify the company, then its founder, then the founder’s education. Each of these links is a hop, and the chain of hops forms a reasoning path through the corpus.
How multi-hop retrieval builds an evidence chain
Multi-hop retrieval works by alternating between retrieval and reasoning until the collected evidence is sufficient to answer. A retriever returns candidate passages for the current query, a reader or language model extracts a bridge entity or intermediate fact, and that fact becomes part of the next query. The loop ends when a stopping criterion is met, such as an answer with sufficient confidence or a fixed maximum number of hops.
The core difficulty is that later hops depend on information that is invisible at query time. Dense retrieval with a single embedding of the original question tends to find passages about the first entity, but rarely the passages about the second or third, because those passages are semantically distant from the question text. Iterative reformulation closes this semantic gap step by step.
The table below shows the stages of one hop and what each stage contributes to the evidence chain.
| Stage | Input | Output | Typical component |
|---|---|---|---|
| Query formulation | Question plus evidence so far | Search query for this hop | Language model or query rewriter |
| Retrieval | Search query | Top-k candidate passages | BM25, dense or hybrid retriever |
| Reranking | Candidate passages | Ordered short list | Cross-encoder or LLM judge |
| Evidence extraction | Short list | Bridge entity or fact | Reader model |
| Sufficiency check | Accumulated evidence | Stop or continue | Classifier or model self-assessment |
Question types that require multiple hops
Multi-hop questions fall into recognizable patterns, and the pattern determines how many hops are needed and whether they can run in parallel. Benchmark datasets such as HotpotQA, 2WikiMultiHopQA and MuSiQue were built around these patterns, and most production questions that need more than one hop fit one of them.
| Pattern | Example structure | Typical hops | Hop order |
|---|---|---|---|
| Bridge | Attribute of an entity named in another document | 2-3 | Sequential |
| Comparison | Which of two entities has the larger value | 2 | Parallel |
| Intersection | Entity satisfying two independent conditions | 2 | Parallel, then merge |
| Temporal chain | State of an entity at the time of another event | 2-4 | Sequential |
| Aggregation | Count or list across many related records | Variable | Fan-out |
Bridge questions are the hardest case for single-pass retrieval because the second hop cannot be predicted from the question alone. Comparison questions are easier: both entities are named, so a system can decompose the question and issue independent queries simultaneously, which reduces latency.
Multi-hop retrieval compared with related approaches
Multi-hop retrieval overlaps with several neighboring techniques, but it is defined by dependency between retrieval steps rather than by any particular index or model. The comparison below clarifies where the boundaries lie.
| Approach | Number of searches | Dependency between searches | Main strength |
|---|---|---|---|
| Single-pass RAG | One | None | Low latency and cost |
| Query expansion | One, with a broadened query | None | Better recall for paraphrases |
| Multi-hop retrieval | Several | Each depends on earlier evidence | Answers compositional questions |
| GraphRAG | Graph traversal plus text search | Follows explicit graph edges | Global and relational queries |
| Agentic RAG | Open-ended | Planned by an agent | Flexible tool use beyond search |
A knowledge graph can make hops explicit, since an edge between two entities is already a stored link. Text-based multi-hop retrieval, by contrast, discovers links at query time from unstructured passages. Agentic RAG often contains multi-hop retrieval as one behavior, but an agent may also call calculators or APIs, so the two terms are not synonyms.
Implementation strategies and design choices
Practitioners implement multi-hop retrieval with one of three broad strategies. Question decomposition asks a language model to split the question into sub-questions up front. Iterative retrieval, as in the IRCoT and Self-Ask methods, interleaves chain-of-thought reasoning with retrieval calls. Learned multi-hop retrievers, such as MDR, train a dense encoder to embed the question together with previously retrieved passages.
- Hop budget: most real questions need two or three hops; a cap of three to five hops prevents runaway loops.
- Top-k per hop: retrieving 3 to 10 passages per hop balances recall against context growth.
- Beam search over paths: keeping several candidate chains alive improves recall when an early hop is ambiguous.
- Evidence memory: storing extracted facts, not full passages, keeps the context window small across hops.
The following decision table helps select a strategy for a given workload.
| Situation | Recommended strategy | Reason |
|---|---|---|
| Question structure is predictable | Decomposition with parallel sub-queries | Lowest latency, easy to audit |
| Later hops depend on unknown entities | Iterative retrieval with reasoning | Next query needs earlier evidence |
| Large labeled dataset is available | Learned multi-hop dense retriever | Fast inference, no LLM per hop |
| Entities and relations are curated | Knowledge graph traversal | Exact links, no retrieval noise |
Failure modes and evaluation of multi-hop systems
The dominant risk is error propagation: if the first hop retrieves the wrong bridge entity, every later hop searches for the wrong thing, and the final answer can be confidently incorrect. A second risk is shortcut reasoning, where a model answers from parametric memory or a single lucky passage without completing the chain, which inflates benchmark scores without real multi-hop ability.
| Symptom | Likely cause | Fix |
|---|---|---|
| Plausible but wrong answer | Incorrect bridge entity at hop one | Beam search and reranking per hop |
| Loop repeats the same query | No novelty check between hops | Deduplicate queries and passages |
| Latency of several seconds | Sequential LLM call per hop | Parallelize independent sub-questions |
| Context window overflow | Full passages kept from every hop | Store extracted facts only |
Evaluation should measure supporting-fact recall for each hop, not only final answer accuracy. Metrics such as exact match and F1 on the answer, combined with recall of the gold supporting passages and mean reciprocal rank per hop, reveal whether the system genuinely followed the chain. Latency and cost per question must be reported as well, because each hop typically adds one retrieval call and often one model call.
The simple explanation
Some questions cannot be answered by looking in one place. Imagine asking a librarian which school the author of a certain novel’s sequel attended. The librarian first finds who wrote the sequel, then looks up that writer’s biography. Each lookup is one step, and the second step is only possible after the first.
Multi-hop retrieval does the same for an AI system. Instead of searching once and hoping the answer appears, it searches, reads, learns a new clue, and searches again with that clue. The main risk is simple: if the first clue is wrong, every later step follows the wrong trail.
How it is used in practice
A pharmaceutical compliance team uses an internal assistant built on a corpus of about 200,000 regulatory filings, trial reports and supplier contracts. An analyst asks which contract manufacturer produced the batches used in the phase III trial of a specific compound. The answer requires three linked documents: the trial report names only a batch code, the batch records name a manufacturing site, and a separate supplier contract links that site to the manufacturer.
A single-pass RAG pipeline returns the trial report and stops, so the assistant cannot answer. After the team adds iterative multi-hop retrieval with a three-hop cap, the system extracts the batch code, retrieves the batch record, extracts the site, and then retrieves the matching contract. Each answer now lists the full evidence chain of three cited passages, and reviewers can verify every link. Median response time rises from about 2 seconds to about 6 seconds, which the team accepts because analysts no longer spend 20 to 30 minutes tracing the same chain by hand.