Semantic textual similarity

Also known as: STS

On this page 7

Semantic textual similarity (STS) is the task of measuring how close two pieces of text are in meaning, expressed as a graded score rather than a yes-or-no label. In artificial intelligence, and specifically in speech & language AI, STS sits underneath search, duplicate detection, retrieval-augmented generation and the evaluation of generated text, because all of them need a reliable answer to one question: do these two sentences say the same thing?

Unlike surface matching, STS rewards shared meaning despite different wording. “The flight was canceled because of fog” and “Fog grounded the plane” share few words but score high, while “The dog bit the man” and “The man bit the dog” share every word yet differ in meaning. Capturing that difference is what makes the task hard.

How semantic textual similarity is scored

STS is scored by comparing a model prediction with human similarity judgments collected on sentence pairs. The classic benchmark convention, established by the SemEval STS shared tasks between 2012 and 2017, uses a continuous scale from 0 to 5, where annotators rate each pair and the average becomes the gold label.

The scale below shows how annotators interpret each level. The middle of the scale is where annotator agreement is weakest, which limits how precisely any model can be evaluated there.

ScoreMeaningTypical pair
5Completely equivalentParaphrases with identical facts
4Mostly equivalentOne unimportant detail differs
3Roughly equivalentImportant detail missing or added
2Not equivalent, some shared detailsSame event, different claims
1Same topic onlyBoth discuss weather, different facts
0UnrelatedNo overlap in topic or content
The 0 to 5 STS annotation scale

Systems are evaluated with correlation against gold scores, not accuracy. Spearman rank correlation is preferred today because it checks whether the model orders pairs correctly, which is what retrieval and ranking need; Pearson correlation is also reported but is sensitive to the shape of the score distribution.

Model architectures for STS

Modern STS relies on neural sentence representations, and two architectures dominate. A bi-encoder turns each sentence into a fixed vector independently, and similarity is the cosine similarity between the two vectors. A cross-encoder reads both sentences together through one transformer and outputs a single score directly.

The difference matters in production. A bi-encoder lets embeddings be computed once and indexed, so millions of comparisons reduce to fast vector arithmetic. A cross-encoder sees word-level interactions between the two texts, which usually raises correlation by a few points, but every pair needs a full forward pass. The table compares the main approaches.

ApproachHow similarity is computedStrengthWeakness
Lexical overlapShared words or n-grams, for example JaccardFast, transparentMisses paraphrases
Averaged word vectorsMean of word2vec or GloVe vectorsCheap, no trainingIgnores word order
Bi-encoderCosine of two sentence embeddingsScales to large corporaCoarser pair interaction
Cross-encoderJoint transformer over both textsHighest correlationSlow, no precomputed index
LLM as judgePrompted model rates the pairHandles nuance, explainsCostly, less consistent
Main STS approaches compared

Bi-encoders are typically trained with contrastive learning: paraphrase pairs are pulled together and unrelated sentences pushed apart in the embedding space. Training on natural language inference data and mined paraphrases produced the jump in quality that made sentence embeddings practical around 2019.

STS is symmetric and graded, and that distinguishes it from neighboring tasks. Textual entailment is directional: a premise may entail a hypothesis without the reverse being true. Paraphrase detection is binary. Semantic relatedness is broader than similarity, since “coffee” and “cup” are related but not similar.

TaskOutputDirectionExample question
Semantic textual similarityGraded scoreSymmetricHow alike are A and B?
Textual entailmentEntail, neutral, contradictA to B onlyDoes A imply B?
Paraphrase detectionYes or noSymmetricDo A and B mean the same?
Semantic relatednessGraded scoreSymmetricAre A and B connected at all?
Semantic searchRanked listQuery to documentsWhich texts answer A?
How STS differs from neighboring tasks

In practice these boundaries blur, because the same embedding model often powers STS, search and clustering. Engineers should still choose the evaluation that matches the use case: a model tuned for question-to-passage retrieval is not automatically good at symmetric sentence comparison.

Where STS is used in practice

STS underpins any system that must match meaning rather than strings. Common applications include deduplicating support tickets, detecting near-duplicate questions in forums, grouping survey answers, matching resumes to job descriptions and evaluating machine translation or summaries against references. In speech pipelines it compares transcribed utterances with known intents or FAQ entries after speech-to-text.

Choosing a setup is mainly a question of volume and required precision. The decision table summarizes typical choices.

SituationRecommended setupReason
Millions of candidate textsBi-encoder with vector indexPrecomputed embeddings scale
Top 50 results need precise orderBi-encoder, then cross-encoder rerankAccuracy where it matters
Specialized domain vocabularyFine-tune on in-domain pairsGeneral models misread jargon
Multiple languages in one corpusMultilingual embedding modelShared vector space across languages
Low volume, explanation neededLLM judge with fixed rubricReadable reasoning per pair
Choosing an STS setup

A similarity threshold converts scores into decisions, and it must be calibrated on labeled samples from the target data. Cosine values are not comparable across models: one model may place unrelated sentences near 0.2, another near 0.6, so a threshold copied from another system is a frequent source of errors.

Limitations and common mistakes

STS models encode topical closeness more reliably than factual agreement. Many embedding models give high similarity to sentences that differ only in negation, numbers or named entities, such as “the dose is 5 mg” and “the dose is 50 mg”, which is dangerous in medical, legal or financial use.

  • Negation blindness: “approved” and “not approved” can score as near-duplicates in bi-encoders.
  • Length mismatch: comparing a short query with a long document dilutes the embedding and lowers scores unpredictably.
  • Benchmark overfitting: high scores on public STS benchmarks do not guarantee quality on domain text.
  • Anisotropy: embeddings crowded into a narrow cone inflate all cosine values, compressing the useful range.
  • Ambiguous gold labels: human agreement caps the achievable correlation, especially for mid-scale pairs.

The standard mitigation is layered checking: use STS to retrieve candidates, then apply a cross-encoder, entailment model or rule-based check for critical facts before acting on the match.

Put simply

Semantic textual similarity is a way for a computer to judge whether two sentences mean the same thing, even when they use different words. Instead of answering only yes or no, it gives a score, a bit like a teacher grading how close a student’s summary is to the original.

The computer turns each sentence into a list of numbers that captures its meaning, then checks how close those lists are. The key point: similar meaning gets a high score, similar wording alone does not. It is good at spotting the same topic, but it can still miss small details such as a changed number or a missing “not”.

In practice

A software company receives about 2,000 support tickets per week, and agents waste time answering the same problem described in different words. The team embeds every new ticket with a bi-encoder and compares it with roughly 40,000 resolved tickets stored in a vector index, retrieving the 20 closest matches in milliseconds.

A cross-encoder then rescores those 20 candidates, and any match above a threshold of 0.82, calibrated on 600 hand-labeled pairs, is shown to the agent with its past solution. After a month, agents report that about a third of new tickets arrive with a usable suggested answer, and duplicate bug reports are merged automatically. Tickets that mention version numbers still go to a human, because the model sometimes treats “fails on 3.2” and “fails on 2.3” as the same issue.