BERTScore is an automatic metric for evaluating generated text that measures semantic similarity between a candidate and a reference by matching their tokens through contextual embeddings from a pretrained transformer encoder. In model evaluation, it addresses a weakness of n-gram metrics such as BLEU and ROUGE, which reward only exact surface matches and miss paraphrases.
Introduced in 2019 and widely used for machine translation, summarization, captioning and dialogue, BERTScore reports precision, recall and F1 at the token level. It is part of the broader artificial intelligence evaluation toolkit and is usually reported alongside other metrics, not as a sole verdict on quality.
How BERTScore matches tokens with embeddings
BERTScore computes similarity in three stages: encoding, greedy matching and aggregation. Both the candidate and the reference are passed through a pretrained encoder such as BERT, RoBERTa or DeBERTa, producing one contextual vector per token that reflects the word and its surrounding sentence.
Each candidate token is then paired with the reference token of highest cosine similarity, and vice versa. This greedy soft alignment lets “automobile” match “car” or “purchased” match “bought”, which an n-gram metric would count as complete misses.
The resulting token-level similarities are averaged into three scores, summarized in the following table.
| Score | How it is computed | What it indicates |
|---|---|---|
| Precision | Mean best match for each candidate token | Whether generated content is supported by the reference |
| Recall | Mean best match for each reference token | Whether reference content is covered by the output |
| F1 | Harmonic mean of precision and recall | Balanced overall similarity, the usual headline figure |
Two optional refinements change the raw values. IDF weighting gives rare, informative tokens more influence than common function words, and baseline rescaling subtracts the typical score of unrelated sentences so results spread over a more readable range.
Configuration choices that change the score
BERTScore is not a single fixed number; its value depends on the encoder, the layer used and the preprocessing. Two papers reporting BERTScore with different settings produce results that cannot be compared directly, which is why the configuration hash printed by the reference implementation should always be reported.
| Setting | Common options | Effect on results |
|---|---|---|
| Encoder model | RoBERTa-large, DeBERTa-xlarge-mnli, multilingual BERT | Stronger encoders correlate better with human judgment |
| Layer selection | A tuned intermediate layer per model | Intermediate layers usually outperform the final layer |
| IDF weighting | On or off | Reduces credit for frequent tokens |
| Baseline rescaling | On or off | Spreads raw scores from about 0.8-0.95 to a wider range |
| Language | Language-specific or multilingual model | Mismatched models lower reliability |
Without rescaling, raw F1 values for related English sentences often sit between about 0.85 and 0.95, so a difference of 0.01 can be meaningful. With rescaling, the same scores spread toward a 0 to 1 range, but they can occasionally become negative for unrelated text.
BERTScore compared with other text metrics
BERTScore belongs to the family of embedding-based metrics, positioned between surface overlap metrics and model-based judges. The comparison below places it among the metrics most often seen in evaluation reports.
| Metric | Basis of comparison | Main strength | Main weakness |
|---|---|---|---|
| BLEU | N-gram precision with brevity penalty | Fast, standard in translation | Misses paraphrases and synonyms |
| ROUGE | N-gram and subsequence recall | Standard in summarization | Rewards lexical copying |
| BERTScore | Contextual embedding similarity | Captures paraphrase and meaning | Weak on factual and logical errors |
| BLEURT and COMET | Models fine-tuned on human ratings | Higher correlation with human judgment | Need training data, less transparent |
| LLM-as-a-judge | Language model grading with a rubric | Flexible, no reference required | Costly, biased, harder to reproduce |
In machine translation shared tasks, BERTScore generally correlates better with human ratings than BLEU, yet trained metrics such as COMET usually correlate better still. Its advantage is that it needs no fine-tuning on human scores and remains easy to reproduce.
Where BERTScore is useful and where it fails
BERTScore works best when a trustworthy reference exists and the main question is whether the output expresses the same meaning in different words. It is less reliable when correctness depends on specific facts, numbers or negation.
- Negation blindness: “the drug is safe” and “the drug is not safe” share almost all tokens and receive a high score despite opposite meaning.
- Factual errors pass easily: replacing one date or name changes only one token, so a hallucinated detail barely lowers the score.
- Reference dependence: when many valid answers exist, a single reference penalizes correct but different outputs.
- Length and truncation: encoders have a maximum input of about 512 tokens, so long documents must be split or are silently truncated.
- Score compression: raw values cluster in a narrow band, which makes small differences look negligible without rescaling.
A practical decision guide for applying BERTScore in an evaluation pipeline follows.
| Evaluation situation | Role of BERTScore | Recommended complement |
|---|---|---|
| Paraphrase or style transfer | Primary similarity signal | Human spot checks |
| Summarization | Secondary metric | ROUGE plus factual consistency checks |
| Machine translation | Supporting metric | COMET and human evaluation |
| Open-ended chat answers | Weak signal only | LLM-as-a-judge or win rate |
| Factual question answering | Not suitable alone | Exact match or F1 on answer spans |
Reporting BERTScore correctly
Correct reporting makes BERTScore results reproducible and comparable. A report should state the encoder model, the layer, whether IDF weighting and baseline rescaling were used, the library version and whether the figure is precision, recall or F1.
Differences between systems should be tested with paired bootstrap resampling or a similar significance test rather than read from averages alone. Teams should also confirm that the metric agrees with human evaluation on a sample before trusting it for decisions such as model selection.
In simple terms
BERTScore checks whether a computer-written sentence means the same thing as a correct example sentence, even when the words differ. Older scores only counted identical words, so “the film was great” and “the movie was excellent” looked almost unrelated.
BERTScore uses a language model that understands word meaning in context, so it sees those two sentences as close. The catch is that it judges similarity, not truth: a sentence with one wrong number can still look almost perfect.
A practical example
A news organization tests two summarization models on 2,000 articles, each with a summary written by an editor. Model A scores ROUGE-L of 0.38 and Model B scores 0.34, so ROUGE alone favors Model A.
The evaluation team adds BERTScore with a DeBERTa encoder and baseline rescaling. Model B reaches a rescaled F1 of 0.42 against 0.39 for Model A, and a paired bootstrap test shows the gap is significant. A review of 100 samples reveals that Model A copies sentences verbatim, while Model B paraphrases accurately and reads more naturally.
Before deployment, the team also runs a factual consistency check and finds that both models occasionally alter figures. The final choice of Model B combines BERTScore, a fact-checking step and human review, because no single metric captured every quality dimension.