BERTScore

On this page 7

BERTScore is an automatic metric for evaluating generated text that measures semantic similarity between a candidate and a reference by matching their tokens through contextual embeddings from a pretrained transformer encoder. In model evaluation, it addresses a weakness of n-gram metrics such as BLEU and ROUGE, which reward only exact surface matches and miss paraphrases.

Introduced in 2019 and widely used for machine translation, summarization, captioning and dialogue, BERTScore reports precision, recall and F1 at the token level. It is part of the broader artificial intelligence evaluation toolkit and is usually reported alongside other metrics, not as a sole verdict on quality.

How BERTScore matches tokens with embeddings

BERTScore computes similarity in three stages: encoding, greedy matching and aggregation. Both the candidate and the reference are passed through a pretrained encoder such as BERT, RoBERTa or DeBERTa, producing one contextual vector per token that reflects the word and its surrounding sentence.

Each candidate token is then paired with the reference token of highest cosine similarity, and vice versa. This greedy soft alignment lets “automobile” match “car” or “purchased” match “bought”, which an n-gram metric would count as complete misses.

The resulting token-level similarities are averaged into three scores, summarized in the following table.

ScoreHow it is computedWhat it indicates
PrecisionMean best match for each candidate tokenWhether generated content is supported by the reference
RecallMean best match for each reference tokenWhether reference content is covered by the output
F1Harmonic mean of precision and recallBalanced overall similarity, the usual headline figure
The three BERTScore components

Two optional refinements change the raw values. IDF weighting gives rare, informative tokens more influence than common function words, and baseline rescaling subtracts the typical score of unrelated sentences so results spread over a more readable range.

Configuration choices that change the score

BERTScore is not a single fixed number; its value depends on the encoder, the layer used and the preprocessing. Two papers reporting BERTScore with different settings produce results that cannot be compared directly, which is why the configuration hash printed by the reference implementation should always be reported.

SettingCommon optionsEffect on results
Encoder modelRoBERTa-large, DeBERTa-xlarge-mnli, multilingual BERTStronger encoders correlate better with human judgment
Layer selectionA tuned intermediate layer per modelIntermediate layers usually outperform the final layer
IDF weightingOn or offReduces credit for frequent tokens
Baseline rescalingOn or offSpreads raw scores from about 0.8-0.95 to a wider range
LanguageLanguage-specific or multilingual modelMismatched models lower reliability
BERTScore settings and their effect

Without rescaling, raw F1 values for related English sentences often sit between about 0.85 and 0.95, so a difference of 0.01 can be meaningful. With rescaling, the same scores spread toward a 0 to 1 range, but they can occasionally become negative for unrelated text.

BERTScore compared with other text metrics

BERTScore belongs to the family of embedding-based metrics, positioned between surface overlap metrics and model-based judges. The comparison below places it among the metrics most often seen in evaluation reports.

MetricBasis of comparisonMain strengthMain weakness
BLEUN-gram precision with brevity penaltyFast, standard in translationMisses paraphrases and synonyms
ROUGEN-gram and subsequence recallStandard in summarizationRewards lexical copying
BERTScoreContextual embedding similarityCaptures paraphrase and meaningWeak on factual and logical errors
BLEURT and COMETModels fine-tuned on human ratingsHigher correlation with human judgmentNeed training data, less transparent
LLM-as-a-judgeLanguage model grading with a rubricFlexible, no reference requiredCostly, biased, harder to reproduce
BERTScore and related evaluation metrics

In machine translation shared tasks, BERTScore generally correlates better with human ratings than BLEU, yet trained metrics such as COMET usually correlate better still. Its advantage is that it needs no fine-tuning on human scores and remains easy to reproduce.

Where BERTScore is useful and where it fails

BERTScore works best when a trustworthy reference exists and the main question is whether the output expresses the same meaning in different words. It is less reliable when correctness depends on specific facts, numbers or negation.

  • Negation blindness: “the drug is safe” and “the drug is not safe” share almost all tokens and receive a high score despite opposite meaning.
  • Factual errors pass easily: replacing one date or name changes only one token, so a hallucinated detail barely lowers the score.
  • Reference dependence: when many valid answers exist, a single reference penalizes correct but different outputs.
  • Length and truncation: encoders have a maximum input of about 512 tokens, so long documents must be split or are silently truncated.
  • Score compression: raw values cluster in a narrow band, which makes small differences look negligible without rescaling.

A practical decision guide for applying BERTScore in an evaluation pipeline follows.

Evaluation situationRole of BERTScoreRecommended complement
Paraphrase or style transferPrimary similarity signalHuman spot checks
SummarizationSecondary metricROUGE plus factual consistency checks
Machine translationSupporting metricCOMET and human evaluation
Open-ended chat answersWeak signal onlyLLM-as-a-judge or win rate
Factual question answeringNot suitable aloneExact match or F1 on answer spans
When to rely on BERTScore

Reporting BERTScore correctly

Correct reporting makes BERTScore results reproducible and comparable. A report should state the encoder model, the layer, whether IDF weighting and baseline rescaling were used, the library version and whether the figure is precision, recall or F1.

Differences between systems should be tested with paired bootstrap resampling or a similar significance test rather than read from averages alone. Teams should also confirm that the metric agrees with human evaluation on a sample before trusting it for decisions such as model selection.

In simple terms

BERTScore checks whether a computer-written sentence means the same thing as a correct example sentence, even when the words differ. Older scores only counted identical words, so “the film was great” and “the movie was excellent” looked almost unrelated.

BERTScore uses a language model that understands word meaning in context, so it sees those two sentences as close. The catch is that it judges similarity, not truth: a sentence with one wrong number can still look almost perfect.

A practical example

A news organization tests two summarization models on 2,000 articles, each with a summary written by an editor. Model A scores ROUGE-L of 0.38 and Model B scores 0.34, so ROUGE alone favors Model A.

The evaluation team adds BERTScore with a DeBERTa encoder and baseline rescaling. Model B reaches a rescaled F1 of 0.42 against 0.39 for Model A, and a paired bootstrap test shows the gap is significant. A review of 100 samples reveals that Model A copies sentences verbatim, while Model B paraphrases accurately and reads more naturally.

Before deployment, the team also runs a factual consistency check and finds that both models occasionally alter figures. The final choice of Model B combines BERTScore, a fact-checking step and human review, because no single metric captured every quality dimension.