Exact match

Also known as: EM

On this page 7

Exact match (EM) is an evaluation metric that counts a model prediction as correct only when it is identical to a reference answer, usually after light text normalization, and reports the share of fully correct predictions across a dataset. In model evaluation for artificial intelligence, it is a standard score for question answering, math word problems and other tasks with short, verifiable answers.

EM is deliberately strict: a prediction earns either 1 or 0 per example, with no partial credit. That strictness makes it easy to compute and hard to dispute, but it also means EM can undercount correct answers that are phrased differently from the reference.

How exact match is computed

The EM score is the number of exactly matching predictions divided by the total number of examples, usually reported as a percentage. When several valid reference answers exist, a prediction scores 1 if it matches any one of the references.

Before comparison, most evaluation harnesses apply a normalization function to both the prediction and the reference. The exact steps vary by benchmark, and results computed with different normalization rules are not directly comparable.

The table lists the normalization steps used in common question answering setups and the mismatches each one removes.

StepExample beforeExample after
LowercasingParisparis
Punctuation removalParis.Paris
Article removalthe Eiffel TowerEiffel Tower
Whitespace collapsingNew YorkNew York
Numeric canonicalization1,000.01000
Typical normalization steps before comparison

For reasoning benchmarks such as GSM8K, an answer extraction step comes first: the harness pulls the final number out of a longer chain of reasoning, and only that extracted value is compared. Errors in extraction are a frequent hidden source of score differences between reports.

Where exact match is used

EM fits tasks where a correct answer has one canonical form or a short list of acceptable forms. It is a poor fit for open-ended generation, where many different outputs can be equally good.

Task typeSuitabilityReason
Extractive question answeringHighAnswers are short spans of a source text
Math word problemsHighThe final answer is a single number
Multiple-choice benchmarksHighThe answer is one option letter
Text-to-SQL and code outputPartialEquivalent queries can differ in text
Summarization and translationLowMany valid outputs exist
Suitability of exact match by task type

In multiple-choice benchmarks such as MMLU, EM on the option letter is effectively the same as accuracy. In text-to-SQL work, practitioners often report EM alongside execution accuracy, which runs both queries and compares the results, because two different SQL strings can return identical data.

EM is usually reported together with softer metrics that award partial credit. The most common companion is the token-level F1 score, which measures word overlap between prediction and reference.

MetricWhat it comparesPartial creditMain limitation
Exact matchWhole normalized stringNoPenalizes valid paraphrases
Token-level F1Overlap of answer wordsYesRewards answers with extra wrong words
BLEU and ROUGEN-gram overlap with referencesYesWeak link to factual correctness
BERTScoreSimilarity of contextual embeddingsYesHarder to interpret and reproduce
LLM-as-a-judgeSemantic correctness judged by a modelConfigurableJudge bias and cost
Exact match and neighboring evaluation metrics

The gap between EM and F1 is itself informative. A large gap suggests that the model usually finds the right content but adds or omits words, while a small gap suggests that errors are mostly genuinely wrong answers rather than formatting differences.

Limitations and common pitfalls

The main weakness of EM is sensitivity to surface form. A generative language model that answers “Barack Obama” when the reference is “Obama” scores zero, even though a human grader would accept it.

  • Verbose answers from chat models fail EM unless the prompt enforces a short output format or the harness extracts the answer.
  • Incomplete reference lists turn correct alternative answers into false errors.
  • Inconsistent normalization across papers makes small score differences meaningless.
  • Strict EM on code or SQL ignores semantic equivalence and underrates capable models.
  • Benchmark contamination can inflate EM when test answers appeared in training data.

A practical safeguard is manual error analysis of a sample of EM failures. If many failures are acceptable paraphrases, the evaluation should add references, loosen normalization, or switch to a semantic grading method.

Choosing between strict and lenient matching

Teams often run several matching levels and report them side by side. The right choice depends on whether the evaluation must reflect strict format compliance or underlying knowledge.

SituationRecommended choiceReason
Output feeds an automated systemStrict EM without normalizationDownstream code needs exact format
Comparing with published benchmark resultsBenchmark’s official EM scriptKeeps numbers comparable
Measuring factual knowledgeNormalized EM plus F1Separates knowledge from formatting
Free-form chat answersSemantic grading or human reviewEM would undercount correct answers
Decision guide for matching strictness

Whatever level is chosen, the evaluation report should state the exact normalization and extraction rules, the number of examples and whether multiple references were allowed. Because EM is a proportion, small test sets produce wide uncertainty: on 200 examples, a difference of two or three points can easily fall within random variation, so confidence intervals or repeated runs are advisable before declaring one model better than another.

EM also works well as a regression test in continuous evaluation. Because the metric is deterministic and cheap, a team can rerun it on every model or prompt change and flag any drop immediately, reserving slower semantic or human grading for the cases where EM changes.

In everyday language

Exact match works like a strict quiz grader who only gives a point when your answer is exactly the same as the answer key. Small things such as capital letters or a final period are usually ignored, but different wording is not.

If the key says “1969” and you write “in the year 1969”, a strict grader may mark it wrong. That makes the score easy to trust but sometimes unfair to answers that are correct in substance.

Seen in the real world

An engineering team at an online retailer builds an assistant that answers customer questions such as order cut-off times and return windows. They assemble 800 test questions with short reference answers and measure the first version: it reaches 52 percent exact match but 78 percent token-level F1.

Error analysis shows that most failures are long, polite sentences that contain the right fact. The team changes the evaluation prompt to request a one-phrase answer and adds an extraction step. EM rises to 81 percent without any change to the model, showing that the earlier gap reflected formatting rather than missing knowledge. They keep EM as the release gate and review the remaining failures by hand each sprint.