SPLADE

On this page 7

SPLADE (Sparse Lexical and Expansion model) is a learned sparse retrieval model that uses a transformer language model to turn a query or document into a weighted bag of vocabulary terms, including relevant terms that never appear in the original text. In the retrieval & knowledge layer of artificial intelligence systems, it sits between classic keyword ranking such as BM25 and dense retrieval with embeddings: the output is a sparse vector that a standard inverted index can store and search, yet the term weights are learned from relevance data.

The model was introduced by researchers at Naver Labs Europe in 2021 and refined in several later versions. Its appeal is practical: it delivers neural-quality relevance while keeping the interpretability and infrastructure of lexical search, which makes it a common component of hybrid search and retrieval-augmented generation pipelines.

How SPLADE builds sparse term vectors

SPLADE produces each vector by passing text through a BERT-style encoder and reading the masked language modeling head, which scores every token position against the entire vocabulary of roughly 30,000 WordPiece tokens. Each score indicates how plausible a vocabulary term is at that position, so the head naturally proposes synonyms and related words.

The model then aggregates these scores into one weight per vocabulary term. It applies a ReLU activation to remove negative values, a logarithmic saturation to keep a few strong terms from dominating, and max pooling across positions (the first version used sum pooling). The result is a vector the size of the vocabulary in which most entries are zero.

Sparsity is not automatic. During training, a FLOPS regularizer penalizes the expected number of floating-point operations in index lookups, which pushes rarely useful weights to zero. The strength of this penalty is the main efficiency knob: a stronger penalty yields shorter vectors and faster search at some cost in effectiveness.

The following table summarizes the processing stages and what each contributes.

StageOperationEffect on the vector
TokenizationWordPiece split of the inputDefines the term space
MLM head scoringLogits over the full vocabulary per tokenProposes expansion terms
Activation and saturationlog(1 + ReLU(logit))Removes negatives, dampens peaks
PoolingMax over token positionsOne weight per vocabulary term
Sparsity regularizationFLOPS penalty during trainingMost weights become zero
Processing stages of a SPLADE encoder

Term weighting and term expansion

SPLADE solves two long-standing weaknesses of lexical retrieval at once. Term weighting replaces fixed statistics such as TF-IDF with context-aware importance, so a word that is central to the meaning of a passage receives a high weight even if it appears only once.

Term expansion addresses the vocabulary mismatch problem, where a relevant document uses different words than the query. A passage about a “heart attack” can receive nonzero weights for “myocardial” and “infarction”, so it matches a clinical query without any manual synonym list. Expansion applies to documents at indexing time and, in the standard configuration, to queries at search time.

Typical vector sizes depend on the regularization strength and the text length. The values below are indicative ranges rather than fixed properties of the model.

Vector typeTypical nonzero termsMain consequence
Document vectorAbout 100-300 for a passageDetermines index size
Query vectorAbout 10-60Determines lookup cost
BM25 query for comparisonOnly the typed wordsFast but literal matching
Indicative vector sizes and their consequences

SPLADE compared with BM25 and dense retrieval

SPLADE belongs to the family of sparse retrieval methods but is trained like a neural ranker. Compared with BM25, it usually improves recall and ranking quality markedly on in-domain benchmarks such as MS MARCO. Compared with dense retrieval using a bi-encoder, it reaches similar in-domain effectiveness and often shows stronger zero-shot generalization on heterogeneous benchmarks such as BEIR, because exact lexical signals remain available.

The comparison below places SPLADE among its closest neighbors in the retrieval stack.

MethodRepresentationIndex typeKey strength
BM25Raw term statisticsInverted indexNo training, very fast
SPLADELearned weights with expansionInverted indexInterpretable neural relevance
Dense bi-encoderOne continuous embeddingVector index such as HNSWStrong semantic matching
ColBERTOne embedding per tokenMulti-vector indexFine-grained late interaction
SPLADE and related first-stage retrievers

SPLADE is not a reranker. It is a first-stage retriever that selects candidates from the full collection; a cross-encoder can still rerank its top results for additional precision.

Variants and deployment choices

Several SPLADE variants trade effectiveness against cost. SPLADE++ adds knowledge distillation from a cross-encoder teacher and training on hard negatives, which raised effectiveness considerably. Document-only SPLADE encodes only the documents and uses plain tokenized queries, which removes the neural encoder from the query path entirely.

The right configuration depends on latency budgets and hardware. The decision table below covers common situations.

SituationRecommended choiceReason
Strict query latency, no GPUDocument-only encodingNo transformer at query time
Maximum first-stage qualityDistilled SPLADE++ modelBest learned relevance signal
Existing Lucene-based searchIndex weights as term impactsReuses current infrastructure
Mixed lexical and semantic needsFuse with a dense retrieverErrors of the two models differ
Choosing a SPLADE configuration

In practice, document weights are often quantized to integers and stored as term impacts in engines such as Lucene, Elasticsearch, OpenSearch or PISA. Results from SPLADE and a dense model are frequently combined with reciprocal rank fusion in hybrid search.

Limitations and common mistakes

SPLADE inherits costs from both worlds. Expanded documents create longer posting lists than BM25, so naive query processing can be several times slower than keyword search unless the vectors are well regularized and the engine uses dynamic pruning. Encoding a large corpus also requires GPU inference at indexing time.

  • Vocabulary lock-in: the model can only express terms in its tokenizer vocabulary, which limits rare names, codes and non-English text unless a multilingual model is used.
  • Domain shift: a model trained on web questions may weight specialized jargon poorly, so fine-tuning on in-domain pairs is often worthwhile.
  • Overreading interpretability: expansion terms are inspectable, but their weights reflect training statistics, not verified semantic truth.
  • Long documents: the encoder input limit of about 512 tokens means long texts must be split by chunking before encoding.

Stripped to the essentials

Classic search engines look for the exact words you typed. SPLADE adds a smart step before that: a language model reads each document and writes down the words it is really about, including helpful words the author never used, with a score for how important each one is.

Search still works the fast, old-fashioned way, by looking words up in an index, like the index at the back of a book. The difference is that this index was written by a reader who understands meaning. The key takeaway: SPLADE gives keyword search some of the understanding of AI without giving up its speed and transparency.

Where it is applied

A legal technology team runs search over about four million court decisions and supports a retrieval-augmented generation assistant for lawyers. Its BM25 system misses decisions that describe the same concept in different wording, while a dense-only prototype misses exact statute numbers and party names.

The team encodes the corpus once with a SPLADE++ model on two GPUs, stores the weights as integer impacts in its existing search cluster, and fuses SPLADE results with dense results using reciprocal rank fusion. A cross-encoder reranks the top 50 candidates before passages reach the language model.

In internal evaluation on a few hundred annotated queries, recall at 100 improved noticeably over BM25, exact citations continued to match, and query latency stayed within the existing budget. The outcome was a hybrid pipeline with fewer missed precedents and no new vector database for the lexical side.