Cosine similarity

On this page 8

Cosine similarity is a measure of how closely two vectors point in the same direction, computed as the cosine of the angle between them. In artificial intelligence, and especially in the retrieval & knowledge layer of modern systems, it is the default score for deciding whether a query embedding and a document embedding express related meaning, regardless of how long either vector is.

The score ranges from -1 to 1: a value near 1 means the vectors are almost parallel, 0 means they are orthogonal and unrelated, and negative values mean they point in opposing directions. Because it ignores magnitude and looks only at orientation, cosine similarity became the workhorse metric of semantic search, retrieval-augmented generation (RAG) and duplicate detection.

How cosine similarity is calculated

Cosine similarity divides the dot product of two vectors by the product of their Euclidean lengths. Written out, the score equals the sum of a_i * b_i over all dimensions, divided by ||a|| * ||b||. The division is what removes the influence of vector magnitude, so a long passage and a short sentence can still score as near neighbors when they discuss the same idea.

A practical consequence follows directly from the formula. When every vector is first normalized to unit length, the denominator becomes 1, and cosine similarity is simply the dot product. Most vector databases exploit this: they normalize embeddings at ingestion time and then use fast inner-product kernels, which saves a square root and two divisions for every comparison across millions of candidates.

The table below shows how the score should be read for typical text embeddings produced by a modern embedding model.

Score rangeGeometric meaningTypical interpretation
0.90 to 1.00Nearly parallel vectorsParaphrase or near-duplicate
0.75 to 0.90Small angleSame topic, strong relevance
0.50 to 0.75Moderate angleLoosely related content
Around 0Orthogonal vectorsNo shared meaning
Below 0Opposing directionsRare with most text models
Reading cosine similarity scores for text embeddings

These bands are model-dependent, not universal. Some embedding models compress all scores into a narrow band such as 0.6 to 0.9, a property known as anisotropy, so an absolute threshold that works for one model can fail badly for another.

Why direction matters more than magnitude

Embedding models encode meaning mostly in the direction of a vector, while the length often reflects incidental factors such as token count, word frequency or training artifacts. Measuring the angle therefore isolates the semantic signal. Two documents about contract law should match even if one has a vector length of 12 and the other a length of 30.

This property is also why cosine similarity predates neural networks. In classic information retrieval, TF-IDF vectors of long documents had larger norms simply because they contained more words. Cosine normalization prevented long documents from dominating the ranking, and the same logic carried over to dense embeddings.

Cosine similarity compared with other vector metrics

Cosine similarity is one of several ways to quantify vector similarity, and the right choice depends on how the embedding model was trained. The following table compares the metrics that vector indexes such as HNSW or IVF most commonly support.

MetricWhat it measuresSensitive to lengthBest fit
Cosine similarityAngle between vectorsNoMost text embedding models
Dot productAngle and magnitude togetherYesModels trained with inner-product loss
Euclidean distanceStraight-line gap between pointsYesImage features, clustering
Manhattan distanceSum of absolute differencesYesSparse or grid-like data
Hamming distanceCount of differing bitsNot applicableBinary embeddings
Common distance and similarity metrics in vector search

For unit-normalized vectors the three main metrics become mathematically equivalent in ranking. Squared Euclidean distance equals 2 - 2 * cos(theta), so sorting by smallest distance gives exactly the same order as sorting by highest cosine. The choice still matters when vectors are not normalized, or when a model deliberately encodes confidence or popularity in vector length, as some recommendation models do.

Where cosine similarity sits in a retrieval pipeline

In a RAG system, cosine similarity is the first-stage relevance score. A bi-encoder turns the user query into a vector, the vector index returns the top candidates by cosine score, and later stages refine that list. The score is fast and scalable but coarse, because the query and document were embedded independently and never interacted.

The table below places cosine scoring among the stages that typically follow it.

StageScoring methodCandidates handledPurpose
Dense retrievalCosine similarity on embeddingsMillions to billionsFast semantic recall
Sparse retrievalBM25 term weightingMillionsExact keyword matching
FusionReciprocal rank fusionHundredsCombine dense and sparse lists
RerankingCross-encoder relevance20 to 200Precise final ordering
DiversificationMaximal marginal relevance10 to 50Reduce redundant passages
Role of cosine similarity in a typical RAG pipeline

Maximal marginal relevance itself reuses cosine similarity twice: once to reward relevance to the query and once to penalize similarity to passages already selected. Approximate nearest neighbor search structures also rely on the metric internally, since graph edges in HNSW connect vectors with high cosine scores.

Practical guidance for using cosine scores

Good results depend less on the formula than on how it is configured. The decision table below summarizes the choices that practitioners make most often.

SituationRecommended choiceReason
Model card specifies cosineNormalize, then use inner productIdentical ranking, faster kernels
Model trained with dot productKeep raw dot productLength carries learned signal
Need a relevance cutoffCalibrate on labeled pairsScore bands differ per model
Mixing two embedding modelsNever compare across modelsVector spaces are incompatible
Exact-term queries such as codesAdd BM25 in hybrid searchEmbeddings blur rare tokens
Configuration decisions for cosine-based retrieval

The most valuable habit is threshold calibration. Instead of assuming that 0.8 means relevant, teams sample a few hundred query-passage pairs, label them, and pick the cutoff that balances precision and recall for that specific model and corpus.

Limitations and common mistakes

Cosine similarity captures topical closeness, not truth or answer quality. A passage that restates the question can outscore the passage that actually answers it, which is why reranking and grounding checks remain necessary.

  • Negation blindness: “the drug is safe” and “the drug is not safe” often score above 0.9.
  • Cross-model comparison: scores from different embedding models live in unrelated spaces.
  • Fixed universal thresholds: anisotropic models make absolute cutoffs unreliable.
  • Chunk size effects: very long chunks average many topics into one diluted vector.
  • High-dimensional concentration: with weak models, scores cluster tightly and separate poorly.

A further subtlety is that a high score signals similarity, not relevance to intent. Query rewriting, HyDE and metadata filtering exist partly to compensate for the gap between what a vector resembles and what a user actually needs.

The short version

Imagine every sentence as an arrow pointing somewhere in a huge space of meanings. Sentences about the same subject point in roughly the same direction, even if one arrow is longer than the other. Cosine similarity simply measures how closely two arrows line up.

A score near 1 means the two texts are about the same thing, a score near 0 means they have nothing in common. Search systems use this number to find the passages most likely to help answer a question, then pass them on for a closer look.

Where it makes a difference

An internal support assistant at a mid-sized software company indexes about 40,000 help-desk articles split into 400-token chunks. Each chunk is embedded with a 768-dimensional model, normalized to unit length and stored in an HNSW index that ranks by inner product, which equals cosine similarity for these vectors.

Early tests used a fixed cutoff of 0.80, and the assistant often replied that it had no relevant material. After the team labeled 300 real questions, they found that this model placed most correct passages between 0.62 and 0.78. Lowering the cutoff to 0.65, combined with BM25 hybrid search and a cross-encoder reranker, raised the share of questions with a correct supporting passage in the top five from roughly 70 percent to about 88 percent, and unanswered tickets fell noticeably within the first month.