A bi-encoder is a neural retrieval architecture that encodes the query and each document independently into fixed-length vectors and scores their relevance with a simple vector similarity function, usually cosine similarity or the dot product. In retrieval and knowledge systems for artificial intelligence, it is the standard engine behind dense retrieval and semantic search, because document vectors can be computed once, stored in a vector index and searched in milliseconds.
The name describes the structure: two encoder passes, often through one shared transformer with tied weights, that never see the query and the document together. That separation is the source of both its speed and scalability and its main weakness, a coarser relevance judgment than models that read both texts jointly.
How a bi-encoder produces and compares embeddings
A bi-encoder turns text into a point in a shared embedding space where semantic closeness corresponds to geometric closeness. A transformer such as a BERT-style encoder reads the input, produces one contextual vector per token, and a pooling step collapses those token vectors into a single embedding, typically of 384 to 1,536 dimensions.
At query time only the query is encoded. The system then compares that single vector with millions of precomputed document vectors through approximate nearest neighbor search, for example an HNSW or IVF index. Because the expensive transformer pass runs once per query, not once per document, latency stays nearly constant as the collection grows.
The following table lists the main components of a bi-encoder pipeline and what each one controls.
| Component | Typical choice | What it affects |
|---|---|---|
| Encoder backbone | BERT, MiniLM, E5 or similar transformer | Quality, speed and maximum input length |
| Pooling | Mean pooling or the CLS token | How token meaning is summarized |
| Normalization | L2 normalization to unit length | Makes dot product equal to cosine similarity |
| Similarity function | Cosine similarity or dot product | Ranking behavior and index compatibility |
| Vector index | HNSW, IVF or exact search | Recall, latency and memory use |
How bi-encoders are trained
Bi-encoders are trained with contrastive learning, which pulls the embeddings of matching query and passage pairs together and pushes non-matching pairs apart. The most common objective is the InfoNCE or multiple negatives ranking loss, where every other passage in the training batch serves as a negative example.
Training quality depends heavily on negative selection. Random negatives are easy to separate and teach little, while hard negatives, passages that look relevant but are not, force the model to learn fine distinctions. Many strong models also use knowledge distillation from a cross-encoder teacher, which transfers some joint-reading accuracy into the faster architecture.
The table below compares typical training signals and their effect on the resulting model.
| Training signal | Source | Effect on quality |
|---|---|---|
| In-batch negatives | Other pairs in the same batch | Cheap baseline; benefits from large batches |
| Mined hard negatives | Top results from BM25 or an earlier model | Sharper distinctions; risk of false negatives |
| Cross-encoder distillation | Teacher scores on query and passage pairs | Higher accuracy at no inference cost |
| Weak supervision | Titles, headings, question and answer pairs | Broad coverage for general-purpose models |
Bi-encoder compared with cross-encoder and late interaction
The bi-encoder sits at one end of a speed versus accuracy spectrum of neural rankers. A cross-encoder feeds the query and the document through the transformer together, so attention can relate every query token to every document token; this yields better relevance scores but requires a full model pass for every candidate. Late interaction models such as ColBERT keep one vector per token and compare them at query time, which sits between the two.
This table summarizes the practical differences among the three architectures.
| Architecture | Document vectors | Query-time cost | Typical role |
|---|---|---|---|
| Bi-encoder | One vector per document, precomputed | One encoder pass plus vector search | First-stage retrieval over millions of items |
| Cross-encoder | None; scores computed jointly | One pass per query and document pair | Reranking the top 20 to 200 candidates |
| Late interaction | One vector per token, precomputed | Token-level matching over candidates | Accurate retrieval with larger storage |
Production systems therefore combine them: a bi-encoder retrieves broadly, often fused with BM25 in hybrid search, and a cross-encoder reranks narrowly. Bi-encoders also differ from sparse retrieval methods such as SPLADE or TF-IDF, which match weighted vocabulary terms rather than dense semantic directions.
Choosing and configuring a bi-encoder
Selecting a bi-encoder means matching domain, language, input length and budget to the task, not simply picking the highest leaderboard score. Benchmark results on general collections often fail to transfer to legal, medical or source-code text, so a small in-domain evaluation set with a metric such as recall at k or mean reciprocal rank is the most reliable guide.
The decision table below maps common situations to a sensible configuration.
| Situation | Recommended choice | Reason |
|---|---|---|
| Large general corpus, tight latency | Compact model with 384 dimensions | Fast encoding and smaller index |
| Specialized vocabulary | Fine-tune on in-domain pairs | General models miss domain synonyms |
| Mixed languages | Multilingual bi-encoder | Aligns queries and documents across languages |
| Storage limits | Matryoshka or binary embeddings | Shorter vectors with modest quality loss |
| Long documents | Chunking before encoding | Encoders truncate input beyond their limit |
Some models are asymmetric and expect prefixes such as query: and passage: on their inputs. Omitting the expected prefix is a frequent cause of silently degraded recall, so the model documentation must be followed exactly.
Limitations and common mistakes
The central limitation is the single-vector bottleneck: all meaning of a passage must fit into one fixed vector, so details such as negation, exact numbers, rare identifiers or multiple subtopics are easily blurred. Exact-match queries for product codes or names are often handled better by lexical retrieval such as BM25.
- Mixing embedding models: query and document vectors from different models or versions are not comparable, so a model change requires re-embedding the whole corpus.
- Ignoring truncation: text beyond the maximum token length, often 512 tokens, is discarded without warning.
- Trusting raw scores: similarity values are relative rankings, not calibrated probabilities, so fixed score thresholds transfer poorly between models.
- Skipping evaluation: domain shift can reduce recall sharply, which only an in-domain test set reveals.
In simple terms
A bi-encoder works like a librarian who writes a short numeric summary card for every book in advance. When someone asks a question, the librarian writes one card for the question and looks for the cards that are most similar, without rereading any books.
This makes searching very fast, even across millions of documents. The trade-off is that a card cannot capture every detail, so the best systems use the bi-encoder to find a shortlist and then let a slower, more careful model check the top candidates.
A practical example
A software company runs an internal assistant that answers employee questions from about 400,000 documentation chunks. The team encodes every chunk once with a 768-dimension bi-encoder and stores the vectors in an HNSW index. Each question is encoded in roughly 10 milliseconds, and the index returns the 100 nearest chunks in a few more milliseconds.
Initial tests showed that questions containing exact error codes were often missed, so the team added BM25 through hybrid search and a cross-encoder that reranks the top 100 results down to 5 for the language model. After fine-tuning the bi-encoder on 20,000 real question and answer pairs, recall at 10 improved noticeably on their internal test set, while total retrieval latency stayed below 150 milliseconds.