Mean reciprocal rank (MRR) is an evaluation metric that scores a ranked result list by the position of the first relevant item, taking the reciprocal of that position and averaging it over many queries. In artificial intelligence work on retrieval & knowledge, it measures how quickly a search engine, retriever or question answering system puts one correct answer near the top.
The metric ranges from 0 to 1, where 1 means the first result was relevant for every query. MRR is popular because it is simple, intuitive and closely matched to tasks in which a user or a language model needs only one good hit, such as known-item search, factoid question answering and the retrieval stage of RAG pipelines.
How mean reciprocal rank is calculated
For each query, the system returns a ranked list, and the evaluator finds the rank of the first relevant document. The reciprocal rank for that query is 1 divided by that rank, and it is 0 when no relevant item appears in the evaluated list. MRR is the arithmetic mean of reciprocal ranks across all queries in the test set.
The steep shape of the reciprocal function is what defines the metric. Moving a relevant result from rank 2 to rank 1 adds 0.5 to the query score, while moving it from rank 10 to rank 9 adds only about 0.011. The table below shows the reciprocal rank values for common positions.
| First relevant rank | Reciprocal rank | Interpretation |
|---|---|---|
| 1 | 1.000 | Ideal result |
| 2 | 0.500 | Half the credit of a top hit |
| 3 | 0.333 | Still visible on a short list |
| 5 | 0.200 | Often below the fold |
| 10 | 0.100 | Barely rewarded |
| Not retrieved | 0.000 | Complete miss |
A worked case makes the averaging clear. With three queries whose first relevant results sit at ranks 1, 3 and not found, the reciprocal ranks are 1, 0.333 and 0, so MRR equals about 0.444.
Variants and cutoffs: MRR@k
Most published results report MRR at a cutoff k, written MRR@10 or MRR@100, which only inspects the top k results. Any query whose first relevant item falls below rank k receives a score of 0. The MS MARCO passage ranking benchmark made MRR@10 a standard figure for comparing neural rerankers and dense retrieval models.
The cutoff should match how the results will actually be consumed. The table lists typical choices and their practical rationale.
| Cutoff | Typical use | Rationale |
|---|---|---|
| MRR@1 | Single-answer systems | Equivalent to top-1 accuracy |
| MRR@5 | RAG context windows | Only a few chunks reach the prompt |
| MRR@10 | Web and passage ranking | Matches one page of results |
| MRR@100 | First-stage retrievers | Measures candidate recall for reranking |
With exactly one relevant item per query, MRR@1 equals precision at 1, and MRR becomes identical to mean average precision. Differences between these metrics appear only when queries have several relevant documents.
Mean reciprocal rank versus other ranking metrics
MRR differs from other information retrieval metrics because it ignores everything after the first hit. That focus is a strength for single-answer tasks and a weakness for tasks that need many relevant documents, so teams usually report it alongside recall at k and NDCG.
| Metric | What it rewards | Uses graded relevance |
|---|---|---|
| Mean reciprocal rank | Early position of the first relevant item | No |
| Recall at k | Share of all relevant items in the top k | No |
| Precision at k | Share of the top k that is relevant | No |
| Mean average precision | Precision across every relevant item | No |
| NDCG at k | Gain discounted by position | Yes |
A name clash causes frequent confusion. Reciprocal rank fusion borrows the same 1 over rank idea but uses it to merge rankings from BM25 and dense retrieval in hybrid search, not to evaluate them. The two belong to the same family of ideas but serve different purposes.
Using MRR to evaluate retrieval and RAG systems
In RAG evaluation, MRR measures whether the retriever surfaces a supporting chunk early enough to reach the language model. A high score indicates that grounding material appears near the top, which matters when only three to five chunks fit in the prompt.
Reliable MRR measurement depends on a labeled query set with known relevant documents. Teams typically build it from search logs, annotated question and answer pairs or synthetic questions generated from the corpus, then check a sample by hand. The table gives practical guidance for different evaluation goals.
| Evaluation goal | Is MRR suitable | Better companion metric |
|---|---|---|
| Known-item or navigational search | Yes, strongly | Success at 1 |
| Factoid question answering | Yes | Exact match of the final answer |
| Multi-hop retrieval | Partly | Recall at k for all hops |
| Literature or legal review | No | Recall and mean average precision |
| Comparing rerankers | Yes, as MRR@10 | NDCG at 10 |
Because MRR is a mean of bounded per-query values, statistical significance should be tested with paired methods such as a paired t-test or bootstrap resampling over queries. Differences of 0.01 or less on a few hundred queries are often noise.
Limitations and common mistakes
The main limitation of MRR is blindness to additional relevant results. Two systems with the same first hit receive the same score even if one returns nine further relevant documents and the other returns none.
- Incomplete relevance labels: unlabeled but relevant documents count as misses, which penalizes systems that find new correct answers.
- Mismatched cutoffs: comparing MRR@10 with MRR@100 across papers produces misleading conclusions.
- Non-linear scale: an MRR of 0.5 does not mean results are half as good as perfect; the average rank cannot be read directly.
- Small query sets: a few dozen queries give unstable estimates.
- Ignored graded relevance: a partly relevant item counts the same as a perfect one.
Mean reciprocal rank remains a fast and interpretable signal for single-answer retrieval, but it should never be the only metric in a production evaluation suite.
The short version
Mean reciprocal rank asks a simple question. How far down the list did you have to look before finding something useful? If the right answer is first, the search scores a full point. If it is second, it scores half a point. If it is third, it scores a third.
Do this for many searches and take the average. A score close to 1 means the right answer is almost always on top. It is like grading a librarian on how quickly the correct book appears on the counter.
Where it makes a difference
An internal support team runs a RAG assistant that answers employee questions from about 40,000 policy and IT documents. Only the top four retrieved chunks are passed to the language model, so a supporting chunk at rank seven is useless in practice.
The team assembles 600 real questions from the help desk log and labels the chunk that answers each one. The baseline BM25 retriever reaches an MRR@10 of 0.52. After adding dense retrieval with reciprocal rank fusion and a cross-encoder reranker, MRR@10 rises to 0.71, and a paired bootstrap test confirms the gain is significant.
The share of questions with a correct chunk in the top four climbs from 61 to 82 percent. Answers with unsupported claims drop noticeably, because the model now receives the right evidence far more often.