The BLEU score (Bilingual Evaluation Understudy) is an automatic metric that measures how closely machine-generated text matches one or more human reference texts by counting overlapping word sequences called n-grams. Introduced in 2002 for machine translation, it became one of the most widely reported metrics in model evaluation, and it remains a standard baseline in artificial intelligence research on text generation.
BLEU produces a value between 0 and 1, usually reported on a scale of 0 to 100. Higher values mean greater overlap with the references, but BLEU measures surface similarity, not meaning or fluency, which is the key to using it correctly.
How the BLEU score is calculated
BLEU combines modified n-gram precision with a brevity penalty. For each n-gram length, usually from 1 to 4 words, it counts how many n-grams in the candidate output also appear in a reference, and divides by the total number of candidate n-grams.
The word modified refers to clipping: each candidate n-gram is credited at most as many times as it appears in any single reference. Without clipping, an output such as “the the the the” would earn perfect unigram precision against any reference that contains “the”.
The four precision values are combined with a geometric mean, normally with equal weights of 0.25. Because precision alone rewards very short outputs, BLEU multiplies the result by a brevity penalty that lowers the score when the total candidate length is shorter than the reference length.
| Component | What it measures | Effect on the score |
|---|---|---|
| Unigram precision | Shared single words | Captures word choice and adequacy |
| Bigram to 4-gram precision | Shared word sequences | Captures word order and fluency |
| Clipping | Maximum credited repeats | Prevents rewarding repetition |
| Geometric mean | Combined precision | Drops to zero if any order has no match |
| Brevity penalty | Candidate versus reference length | Penalizes outputs that are too short |
Corpus-level and sentence-level BLEU
BLEU was designed as a corpus-level metric: n-gram counts and lengths are summed across all sentences in a test set before the ratios are computed. Averaging sentence scores is a different calculation and gives different results.
At the sentence level, BLEU is unstable because a short sentence often has no matching 4-gram, which makes the geometric mean zero. This also explains why corpus-level aggregation is more robust: rare zero counts in single sentences are absorbed by the totals across thousands of segments, so one short sentence cannot collapse the whole score. Smoothing methods add small counts to avoid this, but sentence-level BLEU still correlates poorly with human judgment and should not be used to rank individual outputs.
| BLEU range | Common interpretation | Caveat |
|---|---|---|
| Below 10 | Output is barely usable | Depends on language pair |
| 10 to 30 | Gist is understandable | Many errors remain |
| 30 to 50 | Good to high quality | Varies with reference count |
| Above 50 | Very close to references | Can indicate test set leakage |
These bands are rough conventions, not standards. Scores are only comparable when the test set, tokenization, number of references and language pair are the same, so a score of 35 in one paper may not equal 35 in another.
BLEU compared with other text generation metrics
BLEU belongs to the family of lexical overlap metrics, and newer metrics address its blind spots in different ways. The comparison below covers the alternatives most often reported next to it.
| Metric | Basis | Main strength | Main weakness |
|---|---|---|---|
| BLEU | N-gram precision | Fast, simple, reproducible | Ignores synonyms and meaning |
| ROUGE | N-gram recall | Suits summarization | Also purely lexical |
| chrF | Character n-gram F-score | Handles rich morphology | Still surface-based |
| METEOR | Aligned matches with stems and synonyms | Better recall of meaning | Needs language resources |
| BERTScore | Contextual embedding similarity | Credits paraphrases | Depends on the encoder model |
| COMET | Trained regression on human ratings | High agreement with humans | Less transparent, model-dependent |
Because BLEU is precision-oriented, it rewards outputs that avoid unsupported words, while ROUGE is recall-oriented and rewards covering the reference content. For large language models, many teams now pair BLEU with learned metrics, LLM-as-a-judge or human evaluation.
Reporting BLEU reproducibly
BLEU values are highly sensitive to preprocessing, and differences in tokenization alone can shift scores by more than one point. For that reason, the research community adopted SacreBLEU, a reference implementation that applies standardized tokenization and prints a version signature describing every setting.
- Report corpus-level BLEU on detokenized output, not an average of sentence scores.
- Publish the SacreBLEU signature, including tokenizer, smoothing and number of references.
- Use the same test set and references for every system being compared.
- Treat differences below about one point as uncertain unless confirmed by significance testing such as paired bootstrap resampling.
- Check that test sentences did not appear in training data, since contamination inflates scores.
Limitations of the BLEU score
The core limitation is that BLEU rewards matching words, not correct meaning. A translation that uses valid synonyms or a different but correct word order can score low, while a fluent-looking output with a reversed negation can score high because most n-grams still match.
BLEU also performs poorly for open-ended generation tasks such as dialogue, story writing or code, where many different outputs are equally correct. Its agreement with human ratings has weakened for strong modern systems, whose outputs differ from references mainly in ways BLEU cannot detect. It remains useful for tracking progress within one setup, not for absolute quality claims.
A further weakness concerns reference dependence. With a single reference, BLEU penalizes every legitimate variation the reference writer did not choose, so scores rise noticeably when multiple references are available. Languages with free word order or rich inflection, such as Finnish or Czech, also tend to receive lower BLEU values for translations of comparable quality, which makes cross-language comparisons unreliable.
Explained without jargon
The BLEU score checks how many words and short phrases in a machine translation also appear in a translation written by a person. More shared phrases mean a higher score.
It works like grading an answer by comparing it with an answer key word by word. That is quick and consistent, but it can miss a correct answer phrased differently. A high BLEU score suggests the machine is close to human wording, not that every sentence is right.
A typical use case
A localization team trains a new English-to-German translation model for technical product manuals. To decide whether it should replace the current model, the team translates a held-out test set of 2,000 sentences with both systems and scores them with SacreBLEU against professional reference translations.
The current model scores 38.2 BLEU and the new one scores 41.5 BLEU, a gain that paired bootstrap resampling shows is statistically significant. Because BLEU cannot judge terminology errors that change meaning, the team also has two translators rate 200 sampled sentences. Their ratings confirm the improvement, and the new model is deployed with BLEU kept as a regression check for future updates.