Cross-encoder

On this page 7

A cross-encoder is a transformer model that takes a query and a candidate text as one joint input and produces a single relevance score for that pair. In artificial intelligence, and particularly in retrieval & knowledge systems, it is the standard architecture for reranking: a fast first-stage retriever proposes a few hundred candidates, and the cross-encoder reads each pair in full to decide which passages truly answer the question.

The design trades speed for accuracy. Because every query token can attend to every document token, a cross-encoder captures fine-grained interactions such as negation, word order and exact entity matches that independent embeddings miss. The price is that nothing can be precomputed, so it must run once per query-document pair at search time.

How a cross-encoder scores a query-document pair

A cross-encoder scores relevance by concatenating the query and the passage into one token sequence and passing it through a transformer encoder. In a BERT-style model the input looks like [CLS] query [SEP] passage [SEP], and full self-attention runs across both segments in every layer.

After the final layer, the hidden state of the classification token, or a pooled representation, feeds a small linear head that outputs one number. Depending on training, that number is a logit for binary relevance, a graded score, or a value later passed through a sigmoid to obtain a probability-like confidence.

The process for one query typically follows these steps. The table summarizes the components involved and their role in the final score.

ComponentRoleTypical choice
Joint input sequencePlaces query and passage in one contextQuery plus passage, 256 to 512 tokens
Transformer encoderModels token interactions across both textsBERT, MiniLM, ELECTRA, XLM-R or a small LLM
Scoring headMaps the pooled state to a scalarSingle linear layer
Training objectiveTeaches the ranking behaviorBinary cross-entropy, pairwise or listwise loss
Output scoreOrders candidates for one queryUnbounded logit or 0 to 1 value
Components of a cross-encoder reranker

Scores are comparable within one query but not reliably across different queries, because the model was trained to separate relevant from irrelevant passages for the same question. Treating a raw score as a universal relevance threshold therefore needs calibration on real data.

Cross-encoder versus bi-encoder and late interaction

The cross-encoder differs from a bi-encoder in where the query and document meet. A bi-encoder embeds each text separately into a vector and compares vectors with cosine similarity, so document embeddings can be computed once and indexed. A cross-encoder only sees the texts together, which makes it more accurate and far slower.

Late-interaction models such as ColBERT sit between the two: they store per-token embeddings for documents and compute a cheap interaction at query time. The comparison below shows why production systems usually combine architectures rather than pick one.

ArchitectureInteraction pointPrecomputableRelative accuracyTypical role
Bi-encoderAfter encoding, via vector similarityYes, full document vectorsGoodFirst-stage dense retrieval
Late interaction (ColBERT)Token-level similarity after encodingYes, token vectorsVery goodRetrieval or light reranking
Cross-encoderInside every attention layerNoHighest of the threeSecond-stage reranking
LLM listwise rerankerAcross many passages at onceNoHigh, variableFinal reranking of a short list
Comparison of neural retrieval architectures

Lexical methods such as BM25 and sparse models such as SPLADE also serve as first stages. A hybrid search stage followed by a cross-encoder is a common pattern because the reranker can correct errors from both the lexical and the semantic retriever.

Where cross-encoders sit in a retrieval pipeline

A cross-encoder belongs in the second stage of a retrieve-then-rerank pipeline. The first stage narrows millions of chunks to a candidate set, usually the top 50 to 200 results, and the cross-encoder reorders that set so the best few passages reach the user or the language model.

In retrieval-augmented generation (RAG), this ordering matters because the generator only sees a limited context window. Moving the correct passage from position 30 to position 2 often decides whether the answer is grounded or hallucinated, and it allows a smaller final context of 3 to 10 passages.

The depth of reranking is a tuning decision. The table outlines typical settings and their consequences.

Candidates rerankedLatency impactRecall benefitWhen it fits
20 to 50LowLimited if the first stage is weakInteractive search with strict latency
100 to 200ModerateStrong for most corporaTypical RAG and enterprise search
500 or moreHighDiminishing returnsOffline evaluation or batch jobs
Choosing the reranking depth

A practical rule is to measure recall at the candidate depth before tuning the reranker: if the relevant passage is absent from the candidate set, no cross-encoder can recover it.

Training and adapting a cross-encoder

Cross-encoders are trained on labeled query-passage pairs, most commonly from public passage-ranking datasets such as MS MARCO, and then optionally adapted to a domain. The quality of the negative examples usually matters more than the model size.

  • Hard negatives are passages that a first-stage retriever ranks highly but that are not relevant; they teach the model subtle distinctions.
  • Pointwise training classifies each pair as relevant or not, which is simple and stable.
  • Pairwise and listwise losses optimize the ordering directly and often improve metrics such as NDCG@10 and mean reciprocal rank.
  • Knowledge distillation lets a large cross-encoder teach a bi-encoder, transferring part of its accuracy to the fast first stage.

Domain adaptation is valuable in legal, medical and technical corpora, where vocabulary differs from web text. Even a few thousand in-domain labeled pairs, or synthetic queries generated by a language model and filtered carefully, can yield measurable ranking gains.

Limitations and common mistakes

The main limitation of a cross-encoder is compute cost that grows linearly with the number of candidates and with sequence length. Reranking 100 passages means 100 forward passes per query, which makes GPU inference or a small distilled model the usual choice for real-time systems.

SymptomLikely causeRemedy
Relevant text scored lowPassage truncated beyond the token limitRerank smaller chunks or the best window
High latency spikesToo many candidates or long inputsReduce depth, batch pairs, use a smaller model
Poor domain resultsModel trained only on web queriesFine-tune with in-domain hard negatives
Unstable score thresholdsRaw logits used across queriesCalibrate or rank instead of filter
No gain over first stageCorrect passages missing from candidatesImprove first-stage recall first
Frequent problems with cross-encoder reranking

Another misconception is that a cross-encoder understands documents beyond its window. Most models accept 512 tokens or less for the joint input, so long documents must be chunked or scored by passage, with the document score derived from the best or aggregated chunk score.

In simple terms

A cross-encoder is like a careful second reader. A quick first search pulls out a pile of possibly useful pages. The cross-encoder then reads the question and each page side by side and gives every page a score for how well it answers that exact question.

Because it reads both texts together, it notices details that a quick search misses, such as a word like “not” that flips the meaning. The catch is speed: it reads every pair separately, so it only checks the short list, never the whole library.

A practical example

An internal support assistant at a software company answers employee questions from about 400,000 documentation chunks. A hybrid search of BM25 and a bi-encoder returns the top 100 candidates in roughly 40 milliseconds, but evaluation shows that the correct chunk often sits at position 15 or lower, outside the five chunks passed to the language model.

The team adds a compact cross-encoder fine-tuned on 6,000 labeled internal question-passage pairs with hard negatives. Running on one GPU, it reranks the 100 candidates in about 80 milliseconds. On a held-out test set, the share of questions with the correct chunk in the top five rises from roughly 62 percent to 81 percent, and reviewers record noticeably fewer unsupported answers.