Bi-encoder

On this page 7

A bi-encoder is a neural retrieval architecture that encodes the query and each document independently into fixed-length vectors and scores their relevance with a simple vector similarity function, usually cosine similarity or the dot product. In retrieval and knowledge systems for artificial intelligence, it is the standard engine behind dense retrieval and semantic search, because document vectors can be computed once, stored in a vector index and searched in milliseconds.

The name describes the structure: two encoder passes, often through one shared transformer with tied weights, that never see the query and the document together. That separation is the source of both its speed and scalability and its main weakness, a coarser relevance judgment than models that read both texts jointly.

How a bi-encoder produces and compares embeddings

A bi-encoder turns text into a point in a shared embedding space where semantic closeness corresponds to geometric closeness. A transformer such as a BERT-style encoder reads the input, produces one contextual vector per token, and a pooling step collapses those token vectors into a single embedding, typically of 384 to 1,536 dimensions.

At query time only the query is encoded. The system then compares that single vector with millions of precomputed document vectors through approximate nearest neighbor search, for example an HNSW or IVF index. Because the expensive transformer pass runs once per query, not once per document, latency stays nearly constant as the collection grows.

The following table lists the main components of a bi-encoder pipeline and what each one controls.

ComponentTypical choiceWhat it affects
Encoder backboneBERT, MiniLM, E5 or similar transformerQuality, speed and maximum input length
PoolingMean pooling or the CLS tokenHow token meaning is summarized
NormalizationL2 normalization to unit lengthMakes dot product equal to cosine similarity
Similarity functionCosine similarity or dot productRanking behavior and index compatibility
Vector indexHNSW, IVF or exact searchRecall, latency and memory use
Components of a bi-encoder pipeline

How bi-encoders are trained

Bi-encoders are trained with contrastive learning, which pulls the embeddings of matching query and passage pairs together and pushes non-matching pairs apart. The most common objective is the InfoNCE or multiple negatives ranking loss, where every other passage in the training batch serves as a negative example.

Training quality depends heavily on negative selection. Random negatives are easy to separate and teach little, while hard negatives, passages that look relevant but are not, force the model to learn fine distinctions. Many strong models also use knowledge distillation from a cross-encoder teacher, which transfers some joint-reading accuracy into the faster architecture.

The table below compares typical training signals and their effect on the resulting model.

Training signalSourceEffect on quality
In-batch negativesOther pairs in the same batchCheap baseline; benefits from large batches
Mined hard negativesTop results from BM25 or an earlier modelSharper distinctions; risk of false negatives
Cross-encoder distillationTeacher scores on query and passage pairsHigher accuracy at no inference cost
Weak supervisionTitles, headings, question and answer pairsBroad coverage for general-purpose models
Training signals for bi-encoders

Bi-encoder compared with cross-encoder and late interaction

The bi-encoder sits at one end of a speed versus accuracy spectrum of neural rankers. A cross-encoder feeds the query and the document through the transformer together, so attention can relate every query token to every document token; this yields better relevance scores but requires a full model pass for every candidate. Late interaction models such as ColBERT keep one vector per token and compare them at query time, which sits between the two.

This table summarizes the practical differences among the three architectures.

ArchitectureDocument vectorsQuery-time costTypical role
Bi-encoderOne vector per document, precomputedOne encoder pass plus vector searchFirst-stage retrieval over millions of items
Cross-encoderNone; scores computed jointlyOne pass per query and document pairReranking the top 20 to 200 candidates
Late interactionOne vector per token, precomputedToken-level matching over candidatesAccurate retrieval with larger storage
Neural ranking architectures compared

Production systems therefore combine them: a bi-encoder retrieves broadly, often fused with BM25 in hybrid search, and a cross-encoder reranks narrowly. Bi-encoders also differ from sparse retrieval methods such as SPLADE or TF-IDF, which match weighted vocabulary terms rather than dense semantic directions.

Choosing and configuring a bi-encoder

Selecting a bi-encoder means matching domain, language, input length and budget to the task, not simply picking the highest leaderboard score. Benchmark results on general collections often fail to transfer to legal, medical or source-code text, so a small in-domain evaluation set with a metric such as recall at k or mean reciprocal rank is the most reliable guide.

The decision table below maps common situations to a sensible configuration.

SituationRecommended choiceReason
Large general corpus, tight latencyCompact model with 384 dimensionsFast encoding and smaller index
Specialized vocabularyFine-tune on in-domain pairsGeneral models miss domain synonyms
Mixed languagesMultilingual bi-encoderAligns queries and documents across languages
Storage limitsMatryoshka or binary embeddingsShorter vectors with modest quality loss
Long documentsChunking before encodingEncoders truncate input beyond their limit
Bi-encoder configuration decisions

Some models are asymmetric and expect prefixes such as query: and passage: on their inputs. Omitting the expected prefix is a frequent cause of silently degraded recall, so the model documentation must be followed exactly.

Limitations and common mistakes

The central limitation is the single-vector bottleneck: all meaning of a passage must fit into one fixed vector, so details such as negation, exact numbers, rare identifiers or multiple subtopics are easily blurred. Exact-match queries for product codes or names are often handled better by lexical retrieval such as BM25.

  • Mixing embedding models: query and document vectors from different models or versions are not comparable, so a model change requires re-embedding the whole corpus.
  • Ignoring truncation: text beyond the maximum token length, often 512 tokens, is discarded without warning.
  • Trusting raw scores: similarity values are relative rankings, not calibrated probabilities, so fixed score thresholds transfer poorly between models.
  • Skipping evaluation: domain shift can reduce recall sharply, which only an in-domain test set reveals.

In simple terms

A bi-encoder works like a librarian who writes a short numeric summary card for every book in advance. When someone asks a question, the librarian writes one card for the question and looks for the cards that are most similar, without rereading any books.

This makes searching very fast, even across millions of documents. The trade-off is that a card cannot capture every detail, so the best systems use the bi-encoder to find a shortlist and then let a slower, more careful model check the top candidates.

A practical example

A software company runs an internal assistant that answers employee questions from about 400,000 documentation chunks. The team encodes every chunk once with a 768-dimension bi-encoder and stores the vectors in an HNSW index. Each question is encoded in roughly 10 milliseconds, and the index returns the 100 nearest chunks in a few more milliseconds.

Initial tests showed that questions containing exact error codes were often missed, so the team added BM25 through hybrid search and a cross-encoder that reranks the top 100 results down to 5 for the language model. After fine-tuning the bi-encoder on 20,000 real question and answer pairs, recall at 10 improved noticeably on their internal test set, while total retrieval latency stayed below 150 milliseconds.