BLEU score

Also known as: BLEU · Bilingual Evaluation Understudy

On this page 7

The BLEU score (Bilingual Evaluation Understudy) is an automatic metric that measures how closely machine-generated text matches one or more human reference texts by counting overlapping word sequences called n-grams. Introduced in 2002 for machine translation, it became one of the most widely reported metrics in model evaluation, and it remains a standard baseline in artificial intelligence research on text generation.

BLEU produces a value between 0 and 1, usually reported on a scale of 0 to 100. Higher values mean greater overlap with the references, but BLEU measures surface similarity, not meaning or fluency, which is the key to using it correctly.

How the BLEU score is calculated

BLEU combines modified n-gram precision with a brevity penalty. For each n-gram length, usually from 1 to 4 words, it counts how many n-grams in the candidate output also appear in a reference, and divides by the total number of candidate n-grams.

The word modified refers to clipping: each candidate n-gram is credited at most as many times as it appears in any single reference. Without clipping, an output such as “the the the the” would earn perfect unigram precision against any reference that contains “the”.

The four precision values are combined with a geometric mean, normally with equal weights of 0.25. Because precision alone rewards very short outputs, BLEU multiplies the result by a brevity penalty that lowers the score when the total candidate length is shorter than the reference length.

ComponentWhat it measuresEffect on the score
Unigram precisionShared single wordsCaptures word choice and adequacy
Bigram to 4-gram precisionShared word sequencesCaptures word order and fluency
ClippingMaximum credited repeatsPrevents rewarding repetition
Geometric meanCombined precisionDrops to zero if any order has no match
Brevity penaltyCandidate versus reference lengthPenalizes outputs that are too short
Components of the BLEU formula

Corpus-level and sentence-level BLEU

BLEU was designed as a corpus-level metric: n-gram counts and lengths are summed across all sentences in a test set before the ratios are computed. Averaging sentence scores is a different calculation and gives different results.

At the sentence level, BLEU is unstable because a short sentence often has no matching 4-gram, which makes the geometric mean zero. This also explains why corpus-level aggregation is more robust: rare zero counts in single sentences are absorbed by the totals across thousands of segments, so one short sentence cannot collapse the whole score. Smoothing methods add small counts to avoid this, but sentence-level BLEU still correlates poorly with human judgment and should not be used to rank individual outputs.

BLEU rangeCommon interpretationCaveat
Below 10Output is barely usableDepends on language pair
10 to 30Gist is understandableMany errors remain
30 to 50Good to high qualityVaries with reference count
Above 50Very close to referencesCan indicate test set leakage
Typical BLEU ranges in machine translation

These bands are rough conventions, not standards. Scores are only comparable when the test set, tokenization, number of references and language pair are the same, so a score of 35 in one paper may not equal 35 in another.

BLEU compared with other text generation metrics

BLEU belongs to the family of lexical overlap metrics, and newer metrics address its blind spots in different ways. The comparison below covers the alternatives most often reported next to it.

MetricBasisMain strengthMain weakness
BLEUN-gram precisionFast, simple, reproducibleIgnores synonyms and meaning
ROUGEN-gram recallSuits summarizationAlso purely lexical
chrFCharacter n-gram F-scoreHandles rich morphologyStill surface-based
METEORAligned matches with stems and synonymsBetter recall of meaningNeeds language resources
BERTScoreContextual embedding similarityCredits paraphrasesDepends on the encoder model
COMETTrained regression on human ratingsHigh agreement with humansLess transparent, model-dependent
BLEU and related evaluation metrics

Because BLEU is precision-oriented, it rewards outputs that avoid unsupported words, while ROUGE is recall-oriented and rewards covering the reference content. For large language models, many teams now pair BLEU with learned metrics, LLM-as-a-judge or human evaluation.

Reporting BLEU reproducibly

BLEU values are highly sensitive to preprocessing, and differences in tokenization alone can shift scores by more than one point. For that reason, the research community adopted SacreBLEU, a reference implementation that applies standardized tokenization and prints a version signature describing every setting.

  • Report corpus-level BLEU on detokenized output, not an average of sentence scores.
  • Publish the SacreBLEU signature, including tokenizer, smoothing and number of references.
  • Use the same test set and references for every system being compared.
  • Treat differences below about one point as uncertain unless confirmed by significance testing such as paired bootstrap resampling.
  • Check that test sentences did not appear in training data, since contamination inflates scores.

Limitations of the BLEU score

The core limitation is that BLEU rewards matching words, not correct meaning. A translation that uses valid synonyms or a different but correct word order can score low, while a fluent-looking output with a reversed negation can score high because most n-grams still match.

BLEU also performs poorly for open-ended generation tasks such as dialogue, story writing or code, where many different outputs are equally correct. Its agreement with human ratings has weakened for strong modern systems, whose outputs differ from references mainly in ways BLEU cannot detect. It remains useful for tracking progress within one setup, not for absolute quality claims.

A further weakness concerns reference dependence. With a single reference, BLEU penalizes every legitimate variation the reference writer did not choose, so scores rise noticeably when multiple references are available. Languages with free word order or rich inflection, such as Finnish or Czech, also tend to receive lower BLEU values for translations of comparable quality, which makes cross-language comparisons unreliable.

Explained without jargon

The BLEU score checks how many words and short phrases in a machine translation also appear in a translation written by a person. More shared phrases mean a higher score.

It works like grading an answer by comparing it with an answer key word by word. That is quick and consistent, but it can miss a correct answer phrased differently. A high BLEU score suggests the machine is close to human wording, not that every sentence is right.

A typical use case

A localization team trains a new English-to-German translation model for technical product manuals. To decide whether it should replace the current model, the team translates a held-out test set of 2,000 sentences with both systems and scores them with SacreBLEU against professional reference translations.

The current model scores 38.2 BLEU and the new one scores 41.5 BLEU, a gain that paired bootstrap resampling shows is statistically significant. Because BLEU cannot judge terminology errors that change meaning, the team also has two translators rate 200 sampled sentences. Their ratings confirm the improvement, and the new model is deployed with BLEU kept as a regression check for future updates.