ROUGE

Also known as: Recall-Oriented Understudy for Gisting Evaluation

On this page 7

ROUGE (Recall-Oriented Understudy for Gisting Evaluation) is a family of automatic metrics that measure how much a machine-generated text overlaps with one or more human-written reference texts, counted in shared words, n-grams or word sequences. In model evaluation for natural language processing, ROUGE is the long-standing standard for scoring automatic summarization, and it is also applied to headline generation, question answering and other text generation tasks.

ROUGE was designed as a cheap stand-in for human judgment: instead of asking people to grade every summary, it checks whether the important content of the reference appears in the candidate. Its emphasis on recall reflects that origin, although modern reports usually give the F1 score that balances recall and precision.

How ROUGE measures overlap

ROUGE compares a candidate text with a reference text and counts matching units after light preprocessing such as lowercasing, tokenization and optional stemming. Each variant defines the unit differently, but all compute the same three quantities.

  • Recall: matched units divided by the number of units in the reference, showing how much reference content the candidate covers.
  • Precision: matched units divided by the number of units in the candidate, showing how much of the candidate is relevant.
  • F-measure: the harmonic mean of the two, the value most papers report as the headline ROUGE score.

For example, if a reference summary contains 20 bigrams and a candidate shares 8 of them while containing 16 bigrams in total, ROUGE-2 recall is 0.40, precision is 0.50 and F1 is about 0.44. With multiple references, scores are typically computed against each reference and then combined by taking the maximum or the average, which reduces the penalty for valid wording that one annotator happened not to use.

ROUGE variants and what each captures

The ROUGE family includes several variants, and each one rewards a different kind of similarity. The table below lists the ones found in practice.

VariantMatching unitWhat it rewards
ROUGE-1Single words (unigrams)Content coverage, informativeness
ROUGE-2Word pairs (bigrams)Local fluency and phrase-level agreement
ROUGE-LLongest common subsequenceSentence-level word order without strict adjacency
ROUGE-LsumLongest common subsequence per sentence, unionedSummary-level structure across sentences
ROUGE-WWeighted longest common subsequenceConsecutive matches over scattered ones
ROUGE-S and ROUGE-SUSkip-bigrams, optionally with unigramsWord pairs in order with gaps allowed
Main ROUGE variants

ROUGE-L is popular because it needs no fixed n-gram length: the longest common subsequence rewards words that appear in the same order even when other words sit between them. The distinction between ROUGE-L and ROUGE-Lsum matters, because the two can differ by several points on the same output, and papers that do not say which one they used are hard to compare.

ROUGE compared with other text evaluation metrics

ROUGE is one of several reference-based metrics, and it is often reported next to related measures. The comparison below shows how it differs from its closest neighbors in evaluation practice.

MetricBasis of comparisonTypical taskMain weakness
ROUGEN-gram and subsequence overlap, recall-orientedSummarizationIgnores paraphrase and factual accuracy
BLEU scoreN-gram precision with brevity penaltyMachine translationPenalizes valid alternative wording
BERTScoreContextual embedding similaritySummarization, generationDepends on the chosen encoder model
LLM-as-a-judgeModel-written quality assessmentOpen-ended generationBias, cost and limited reproducibility
Human evaluationExpert or crowd ratingsAny generation taskSlow, expensive, requires agreement checks
ROUGE and related evaluation metrics

The contrast with BLEU is the most instructive. BLEU asks how much of the candidate is supported by the reference, which suits translation, where the output should not add content. ROUGE asks how much of the reference is recovered, which suits summarization, where missing key content is the main failure.

Reporting ROUGE scores correctly

ROUGE results are only comparable when the scoring configuration is the same, and small implementation choices can shift scores noticeably. Careful reports state the following details.

  • Which variants were used, usually ROUGE-1, ROUGE-2 and ROUGE-L or ROUGE-Lsum.
  • Whether F1, recall or precision is reported.
  • Whether stemming was applied and how text was tokenized.
  • The implementation used, such as the original Perl package or a reimplementation, since they can disagree slightly.
  • Any length limits on summaries, because longer outputs inflate recall.

Typical values help interpret results. On standard news summarization datasets, strong systems reach ROUGE-1 F1 around 0.40 to 0.47 and ROUGE-2 F1 around 0.18 to 0.24; absolute values depend heavily on the dataset, the number of references and the configuration, so scores should be compared only within one benchmark.

Limitations and misuse of ROUGE

ROUGE measures surface overlap, not meaning, and this produces predictable blind spots. The table below links common symptoms to their cause and a practical response.

SymptomCauseBetter practice
Good paraphrase scores lowNo credit for synonymsAdd BERTScore or human evaluation
Fluent but false summary scores highFactual errors are invisibleRun factual consistency checks
Longer outputs winRecall rewards lengthReport F1 and control length
Scores differ between papersInconsistent configurationUse one documented implementation
Abstractive LLM output looks weakWording diverges from referenceCombine ROUGE with judgment-based metrics
Common ROUGE pitfalls

With large language models, the correlation with human preference has weakened further, because models produce fluent abstractive text that rarely copies reference wording. ROUGE remains useful as a cheap regression check and a sanity signal, but it should not be the sole basis for claiming that one system summarizes better than another.

In plain words

ROUGE is a way to grade a computer-written summary by comparing it with a summary written by a person. It simply counts how many of the same words and phrases appear in both. The more of the human summary the computer manages to repeat, the higher the score.

It is a bit like checking a student’s notes against the teacher’s answer key by looking for matching keywords. That is quick and useful, but a student who explains the same idea in different words may lose points, and a student who copies the right words in a wrong statement may still score well. That is why ROUGE is a helpful first check, not a final verdict.

A real-world example

A legal technology team builds a system that summarizes court decisions into short abstracts for attorneys. It holds out 500 annotated decisions with abstracts written by staff lawyers and scores each new model version with ROUGE-1, ROUGE-2 and ROUGE-Lsum F1, using stemming and a 150-word length limit.

A new version raises ROUGE-2 from 0.19 to 0.23, so the team runs a second check before release. Lawyers review 60 randomly selected abstracts and find that the improved version also cites holdings more accurately, but a few summaries misstate procedural dates despite high overlap scores. The team ships the model and adds an automated date-consistency check, keeping ROUGE as a fast nightly regression signal rather than the only quality gate.