A cross-encoder is a transformer model that takes a query and a candidate text as one joint input and produces a single relevance score for that pair. In artificial intelligence, and particularly in retrieval & knowledge systems, it is the standard architecture for reranking: a fast first-stage retriever proposes a few hundred candidates, and the cross-encoder reads each pair in full to decide which passages truly answer the question.
The design trades speed for accuracy. Because every query token can attend to every document token, a cross-encoder captures fine-grained interactions such as negation, word order and exact entity matches that independent embeddings miss. The price is that nothing can be precomputed, so it must run once per query-document pair at search time.
How a cross-encoder scores a query-document pair
A cross-encoder scores relevance by concatenating the query and the passage into one token sequence and passing it through a transformer encoder. In a BERT-style model the input looks like [CLS] query [SEP] passage [SEP], and full self-attention runs across both segments in every layer.
After the final layer, the hidden state of the classification token, or a pooled representation, feeds a small linear head that outputs one number. Depending on training, that number is a logit for binary relevance, a graded score, or a value later passed through a sigmoid to obtain a probability-like confidence.
The process for one query typically follows these steps. The table summarizes the components involved and their role in the final score.
| Component | Role | Typical choice |
|---|---|---|
| Joint input sequence | Places query and passage in one context | Query plus passage, 256 to 512 tokens |
| Transformer encoder | Models token interactions across both texts | BERT, MiniLM, ELECTRA, XLM-R or a small LLM |
| Scoring head | Maps the pooled state to a scalar | Single linear layer |
| Training objective | Teaches the ranking behavior | Binary cross-entropy, pairwise or listwise loss |
| Output score | Orders candidates for one query | Unbounded logit or 0 to 1 value |
Scores are comparable within one query but not reliably across different queries, because the model was trained to separate relevant from irrelevant passages for the same question. Treating a raw score as a universal relevance threshold therefore needs calibration on real data.
Cross-encoder versus bi-encoder and late interaction
The cross-encoder differs from a bi-encoder in where the query and document meet. A bi-encoder embeds each text separately into a vector and compares vectors with cosine similarity, so document embeddings can be computed once and indexed. A cross-encoder only sees the texts together, which makes it more accurate and far slower.
Late-interaction models such as ColBERT sit between the two: they store per-token embeddings for documents and compute a cheap interaction at query time. The comparison below shows why production systems usually combine architectures rather than pick one.
| Architecture | Interaction point | Precomputable | Relative accuracy | Typical role |
|---|---|---|---|---|
| Bi-encoder | After encoding, via vector similarity | Yes, full document vectors | Good | First-stage dense retrieval |
| Late interaction (ColBERT) | Token-level similarity after encoding | Yes, token vectors | Very good | Retrieval or light reranking |
| Cross-encoder | Inside every attention layer | No | Highest of the three | Second-stage reranking |
| LLM listwise reranker | Across many passages at once | No | High, variable | Final reranking of a short list |
Lexical methods such as BM25 and sparse models such as SPLADE also serve as first stages. A hybrid search stage followed by a cross-encoder is a common pattern because the reranker can correct errors from both the lexical and the semantic retriever.
Where cross-encoders sit in a retrieval pipeline
A cross-encoder belongs in the second stage of a retrieve-then-rerank pipeline. The first stage narrows millions of chunks to a candidate set, usually the top 50 to 200 results, and the cross-encoder reorders that set so the best few passages reach the user or the language model.
In retrieval-augmented generation (RAG), this ordering matters because the generator only sees a limited context window. Moving the correct passage from position 30 to position 2 often decides whether the answer is grounded or hallucinated, and it allows a smaller final context of 3 to 10 passages.
The depth of reranking is a tuning decision. The table outlines typical settings and their consequences.
| Candidates reranked | Latency impact | Recall benefit | When it fits |
|---|---|---|---|
| 20 to 50 | Low | Limited if the first stage is weak | Interactive search with strict latency |
| 100 to 200 | Moderate | Strong for most corpora | Typical RAG and enterprise search |
| 500 or more | High | Diminishing returns | Offline evaluation or batch jobs |
A practical rule is to measure recall at the candidate depth before tuning the reranker: if the relevant passage is absent from the candidate set, no cross-encoder can recover it.
Training and adapting a cross-encoder
Cross-encoders are trained on labeled query-passage pairs, most commonly from public passage-ranking datasets such as MS MARCO, and then optionally adapted to a domain. The quality of the negative examples usually matters more than the model size.
- Hard negatives are passages that a first-stage retriever ranks highly but that are not relevant; they teach the model subtle distinctions.
- Pointwise training classifies each pair as relevant or not, which is simple and stable.
- Pairwise and listwise losses optimize the ordering directly and often improve metrics such as NDCG@10 and mean reciprocal rank.
- Knowledge distillation lets a large cross-encoder teach a bi-encoder, transferring part of its accuracy to the fast first stage.
Domain adaptation is valuable in legal, medical and technical corpora, where vocabulary differs from web text. Even a few thousand in-domain labeled pairs, or synthetic queries generated by a language model and filtered carefully, can yield measurable ranking gains.
Limitations and common mistakes
The main limitation of a cross-encoder is compute cost that grows linearly with the number of candidates and with sequence length. Reranking 100 passages means 100 forward passes per query, which makes GPU inference or a small distilled model the usual choice for real-time systems.
| Symptom | Likely cause | Remedy |
|---|---|---|
| Relevant text scored low | Passage truncated beyond the token limit | Rerank smaller chunks or the best window |
| High latency spikes | Too many candidates or long inputs | Reduce depth, batch pairs, use a smaller model |
| Poor domain results | Model trained only on web queries | Fine-tune with in-domain hard negatives |
| Unstable score thresholds | Raw logits used across queries | Calibrate or rank instead of filter |
| No gain over first stage | Correct passages missing from candidates | Improve first-stage recall first |
Another misconception is that a cross-encoder understands documents beyond its window. Most models accept 512 tokens or less for the joint input, so long documents must be chunked or scored by passage, with the document score derived from the best or aggregated chunk score.
In simple terms
A cross-encoder is like a careful second reader. A quick first search pulls out a pile of possibly useful pages. The cross-encoder then reads the question and each page side by side and gives every page a score for how well it answers that exact question.
Because it reads both texts together, it notices details that a quick search misses, such as a word like “not” that flips the meaning. The catch is speed: it reads every pair separately, so it only checks the short list, never the whole library.
A practical example
An internal support assistant at a software company answers employee questions from about 400,000 documentation chunks. A hybrid search of BM25 and a bi-encoder returns the top 100 candidates in roughly 40 milliseconds, but evaluation shows that the correct chunk often sits at position 15 or lower, outside the five chunks passed to the language model.
The team adds a compact cross-encoder fine-tuned on 6,000 labeled internal question-passage pairs with hard negatives. Running on one GPU, it reranks the 100 candidates in about 80 milliseconds. On a held-out test set, the share of questions with the correct chunk in the top five rises from roughly 62 percent to 81 percent, and reviewers record noticeably fewer unsupported answers.