Semantic textual similarity (STS) is the task of measuring how close two pieces of text are in meaning, expressed as a graded score rather than a yes-or-no label. In artificial intelligence, and specifically in speech & language AI, STS sits underneath search, duplicate detection, retrieval-augmented generation and the evaluation of generated text, because all of them need a reliable answer to one question: do these two sentences say the same thing?
Unlike surface matching, STS rewards shared meaning despite different wording. “The flight was canceled because of fog” and “Fog grounded the plane” share few words but score high, while “The dog bit the man” and “The man bit the dog” share every word yet differ in meaning. Capturing that difference is what makes the task hard.
How semantic textual similarity is scored
STS is scored by comparing a model prediction with human similarity judgments collected on sentence pairs. The classic benchmark convention, established by the SemEval STS shared tasks between 2012 and 2017, uses a continuous scale from 0 to 5, where annotators rate each pair and the average becomes the gold label.
The scale below shows how annotators interpret each level. The middle of the scale is where annotator agreement is weakest, which limits how precisely any model can be evaluated there.
| Score | Meaning | Typical pair |
|---|---|---|
| 5 | Completely equivalent | Paraphrases with identical facts |
| 4 | Mostly equivalent | One unimportant detail differs |
| 3 | Roughly equivalent | Important detail missing or added |
| 2 | Not equivalent, some shared details | Same event, different claims |
| 1 | Same topic only | Both discuss weather, different facts |
| 0 | Unrelated | No overlap in topic or content |
Systems are evaluated with correlation against gold scores, not accuracy. Spearman rank correlation is preferred today because it checks whether the model orders pairs correctly, which is what retrieval and ranking need; Pearson correlation is also reported but is sensitive to the shape of the score distribution.
Model architectures for STS
Modern STS relies on neural sentence representations, and two architectures dominate. A bi-encoder turns each sentence into a fixed vector independently, and similarity is the cosine similarity between the two vectors. A cross-encoder reads both sentences together through one transformer and outputs a single score directly.
The difference matters in production. A bi-encoder lets embeddings be computed once and indexed, so millions of comparisons reduce to fast vector arithmetic. A cross-encoder sees word-level interactions between the two texts, which usually raises correlation by a few points, but every pair needs a full forward pass. The table compares the main approaches.
| Approach | How similarity is computed | Strength | Weakness |
|---|---|---|---|
| Lexical overlap | Shared words or n-grams, for example Jaccard | Fast, transparent | Misses paraphrases |
| Averaged word vectors | Mean of word2vec or GloVe vectors | Cheap, no training | Ignores word order |
| Bi-encoder | Cosine of two sentence embeddings | Scales to large corpora | Coarser pair interaction |
| Cross-encoder | Joint transformer over both texts | Highest correlation | Slow, no precomputed index |
| LLM as judge | Prompted model rates the pair | Handles nuance, explains | Costly, less consistent |
Bi-encoders are typically trained with contrastive learning: paraphrase pairs are pulled together and unrelated sentences pushed apart in the embedding space. Training on natural language inference data and mined paraphrases produced the jump in quality that made sentence embeddings practical around 2019.
STS compared with related language tasks
STS is symmetric and graded, and that distinguishes it from neighboring tasks. Textual entailment is directional: a premise may entail a hypothesis without the reverse being true. Paraphrase detection is binary. Semantic relatedness is broader than similarity, since “coffee” and “cup” are related but not similar.
| Task | Output | Direction | Example question |
|---|---|---|---|
| Semantic textual similarity | Graded score | Symmetric | How alike are A and B? |
| Textual entailment | Entail, neutral, contradict | A to B only | Does A imply B? |
| Paraphrase detection | Yes or no | Symmetric | Do A and B mean the same? |
| Semantic relatedness | Graded score | Symmetric | Are A and B connected at all? |
| Semantic search | Ranked list | Query to documents | Which texts answer A? |
In practice these boundaries blur, because the same embedding model often powers STS, search and clustering. Engineers should still choose the evaluation that matches the use case: a model tuned for question-to-passage retrieval is not automatically good at symmetric sentence comparison.
Where STS is used in practice
STS underpins any system that must match meaning rather than strings. Common applications include deduplicating support tickets, detecting near-duplicate questions in forums, grouping survey answers, matching resumes to job descriptions and evaluating machine translation or summaries against references. In speech pipelines it compares transcribed utterances with known intents or FAQ entries after speech-to-text.
Choosing a setup is mainly a question of volume and required precision. The decision table summarizes typical choices.
| Situation | Recommended setup | Reason |
|---|---|---|
| Millions of candidate texts | Bi-encoder with vector index | Precomputed embeddings scale |
| Top 50 results need precise order | Bi-encoder, then cross-encoder rerank | Accuracy where it matters |
| Specialized domain vocabulary | Fine-tune on in-domain pairs | General models misread jargon |
| Multiple languages in one corpus | Multilingual embedding model | Shared vector space across languages |
| Low volume, explanation needed | LLM judge with fixed rubric | Readable reasoning per pair |
A similarity threshold converts scores into decisions, and it must be calibrated on labeled samples from the target data. Cosine values are not comparable across models: one model may place unrelated sentences near 0.2, another near 0.6, so a threshold copied from another system is a frequent source of errors.
Limitations and common mistakes
STS models encode topical closeness more reliably than factual agreement. Many embedding models give high similarity to sentences that differ only in negation, numbers or named entities, such as “the dose is 5 mg” and “the dose is 50 mg”, which is dangerous in medical, legal or financial use.
- Negation blindness: “approved” and “not approved” can score as near-duplicates in bi-encoders.
- Length mismatch: comparing a short query with a long document dilutes the embedding and lowers scores unpredictably.
- Benchmark overfitting: high scores on public STS benchmarks do not guarantee quality on domain text.
- Anisotropy: embeddings crowded into a narrow cone inflate all cosine values, compressing the useful range.
- Ambiguous gold labels: human agreement caps the achievable correlation, especially for mid-scale pairs.
The standard mitigation is layered checking: use STS to retrieve candidates, then apply a cross-encoder, entailment model or rule-based check for critical facts before acting on the match.
Put simply
Semantic textual similarity is a way for a computer to judge whether two sentences mean the same thing, even when they use different words. Instead of answering only yes or no, it gives a score, a bit like a teacher grading how close a student’s summary is to the original.
The computer turns each sentence into a list of numbers that captures its meaning, then checks how close those lists are. The key point: similar meaning gets a high score, similar wording alone does not. It is good at spotting the same topic, but it can still miss small details such as a changed number or a missing “not”.
In practice
A software company receives about 2,000 support tickets per week, and agents waste time answering the same problem described in different words. The team embeds every new ticket with a bi-encoder and compares it with roughly 40,000 resolved tickets stored in a vector index, retrieving the 20 closest matches in milliseconds.
A cross-encoder then rescores those 20 candidates, and any match above a threshold of 0.82, calibrated on 600 hand-labeled pairs, is shown to the agent with its past solution. After a month, agents report that about a third of new tickets arrive with a usable suggested answer, and duplicate bug reports are merged automatically. Tickets that mention version numbers still go to a human, because the model sometimes treats “fails on 3.2” and “fails on 2.3” as the same issue.