Exact match (EM) is an evaluation metric that counts a model prediction as correct only when it is identical to a reference answer, usually after light text normalization, and reports the share of fully correct predictions across a dataset. In model evaluation for artificial intelligence, it is a standard score for question answering, math word problems and other tasks with short, verifiable answers.
EM is deliberately strict: a prediction earns either 1 or 0 per example, with no partial credit. That strictness makes it easy to compute and hard to dispute, but it also means EM can undercount correct answers that are phrased differently from the reference.
How exact match is computed
The EM score is the number of exactly matching predictions divided by the total number of examples, usually reported as a percentage. When several valid reference answers exist, a prediction scores 1 if it matches any one of the references.
Before comparison, most evaluation harnesses apply a normalization function to both the prediction and the reference. The exact steps vary by benchmark, and results computed with different normalization rules are not directly comparable.
The table lists the normalization steps used in common question answering setups and the mismatches each one removes.
| Step | Example before | Example after |
|---|---|---|
| Lowercasing | Paris | paris |
| Punctuation removal | Paris. | Paris |
| Article removal | the Eiffel Tower | Eiffel Tower |
| Whitespace collapsing | New York | New York |
| Numeric canonicalization | 1,000.0 | 1000 |
For reasoning benchmarks such as GSM8K, an answer extraction step comes first: the harness pulls the final number out of a longer chain of reasoning, and only that extracted value is compared. Errors in extraction are a frequent hidden source of score differences between reports.
Where exact match is used
EM fits tasks where a correct answer has one canonical form or a short list of acceptable forms. It is a poor fit for open-ended generation, where many different outputs can be equally good.
| Task type | Suitability | Reason |
|---|---|---|
| Extractive question answering | High | Answers are short spans of a source text |
| Math word problems | High | The final answer is a single number |
| Multiple-choice benchmarks | High | The answer is one option letter |
| Text-to-SQL and code output | Partial | Equivalent queries can differ in text |
| Summarization and translation | Low | Many valid outputs exist |
In multiple-choice benchmarks such as MMLU, EM on the option letter is effectively the same as accuracy. In text-to-SQL work, practitioners often report EM alongside execution accuracy, which runs both queries and compares the results, because two different SQL strings can return identical data.
Exact match compared with related metrics
EM is usually reported together with softer metrics that award partial credit. The most common companion is the token-level F1 score, which measures word overlap between prediction and reference.
| Metric | What it compares | Partial credit | Main limitation |
|---|---|---|---|
| Exact match | Whole normalized string | No | Penalizes valid paraphrases |
| Token-level F1 | Overlap of answer words | Yes | Rewards answers with extra wrong words |
| BLEU and ROUGE | N-gram overlap with references | Yes | Weak link to factual correctness |
| BERTScore | Similarity of contextual embeddings | Yes | Harder to interpret and reproduce |
| LLM-as-a-judge | Semantic correctness judged by a model | Configurable | Judge bias and cost |
The gap between EM and F1 is itself informative. A large gap suggests that the model usually finds the right content but adds or omits words, while a small gap suggests that errors are mostly genuinely wrong answers rather than formatting differences.
Limitations and common pitfalls
The main weakness of EM is sensitivity to surface form. A generative language model that answers “Barack Obama” when the reference is “Obama” scores zero, even though a human grader would accept it.
- Verbose answers from chat models fail EM unless the prompt enforces a short output format or the harness extracts the answer.
- Incomplete reference lists turn correct alternative answers into false errors.
- Inconsistent normalization across papers makes small score differences meaningless.
- Strict EM on code or SQL ignores semantic equivalence and underrates capable models.
- Benchmark contamination can inflate EM when test answers appeared in training data.
A practical safeguard is manual error analysis of a sample of EM failures. If many failures are acceptable paraphrases, the evaluation should add references, loosen normalization, or switch to a semantic grading method.
Choosing between strict and lenient matching
Teams often run several matching levels and report them side by side. The right choice depends on whether the evaluation must reflect strict format compliance or underlying knowledge.
| Situation | Recommended choice | Reason |
|---|---|---|
| Output feeds an automated system | Strict EM without normalization | Downstream code needs exact format |
| Comparing with published benchmark results | Benchmark’s official EM script | Keeps numbers comparable |
| Measuring factual knowledge | Normalized EM plus F1 | Separates knowledge from formatting |
| Free-form chat answers | Semantic grading or human review | EM would undercount correct answers |
Whatever level is chosen, the evaluation report should state the exact normalization and extraction rules, the number of examples and whether multiple references were allowed. Because EM is a proportion, small test sets produce wide uncertainty: on 200 examples, a difference of two or three points can easily fall within random variation, so confidence intervals or repeated runs are advisable before declaring one model better than another.
EM also works well as a regression test in continuous evaluation. Because the metric is deterministic and cheap, a team can rerun it on every model or prompt change and flag any drop immediately, reserving slower semantic or human grading for the cases where EM changes.
In everyday language
Exact match works like a strict quiz grader who only gives a point when your answer is exactly the same as the answer key. Small things such as capital letters or a final period are usually ignored, but different wording is not.
If the key says “1969” and you write “in the year 1969”, a strict grader may mark it wrong. That makes the score easy to trust but sometimes unfair to answers that are correct in substance.
Seen in the real world
An engineering team at an online retailer builds an assistant that answers customer questions such as order cut-off times and return windows. They assemble 800 test questions with short reference answers and measure the first version: it reaches 52 percent exact match but 78 percent token-level F1.
Error analysis shows that most failures are long, polite sentences that contain the right fact. The team changes the evaluation prompt to request a one-phrase answer and adds an extraction step. EM rises to 81 percent without any change to the model, showing that the earlier gap reflected formatting rather than missing knowledge. They keep EM as the release gate and review the remaining failures by hand each sprint.