Encoder-only model

Also known as: encoder-only transformer

On this page 7

An encoder-only model is a transformer architecture that keeps only the encoder stack and reads every token of an input with bidirectional self-attention, producing one contextual vector per token instead of generating new text. Among language models in artificial intelligence, it is the family behind BERT, RoBERTa, DeBERTa and many sentence-embedding models, and it is also called an encoder-only transformer.

Encoder-only models are built for understanding rather than generation. They power classification, named entity recognition, semantic search, reranking and extractive question answering, where the goal is to represent or label existing text accurately, cheaply and quickly.

How an encoder-only model processes text

An encoder-only model turns a token sequence into a sequence of contextual embeddings of the same length. Text is first split by a subword tokenizer such as WordPiece or a byte-level BPE variant, each token is mapped to a learned vector, and position information is added through absolute embeddings or rotary encodings.

The vectors then pass through a stack of identical layers, typically 12 to 24 layers in common checkpoints. Each layer combines multi-head self-attention with a position-wise feed-forward network, residual connections and layer normalization. The defining detail is the absence of a causal mask: every token can attend to every other token, both to its left and to its right, so the representation of a word reflects its full sentence context.

Because no token is ever predicted from its own future, the output is not a probability distribution over the next word. Instead, downstream components read the final hidden states. A special [CLS] token or a mean pooling step summarizes the whole input for classification and retrieval, while per-token vectors feed tagging and span extraction heads.

The following table lists the main components of a typical encoder-only model and the role each plays.

ComponentTypical settingRole
TokenizerWordPiece or BPE, 30k to 250k entriesSplits text into subword units
Hidden size768 (base) or 1024 (large)Width of each token vector
Attention heads12 to 16 per layerCapture different token relations
Maximum sequence length512 tokens, newer designs 8,192Limits the input per pass
Pooling strategy[CLS] token or mean poolingBuilds one vector per input
Core components of an encoder-only model

Pretraining objectives for encoder-only models

Encoder-only models are pretrained with denoising objectives rather than next-token prediction, because bidirectional attention would let the model simply copy the answer. The best-known objective is masked language modeling (MLM), which hides roughly 15 percent of tokens and trains the model to recover them from the surrounding context, the approach that defines a masked language model.

Later research refined this recipe. RoBERTa showed that dropping next sentence prediction, training longer and using dynamic masking improved results. ELECTRA replaced masking with replaced token detection, in which a small generator corrupts tokens and the encoder learns to spot every substitution, giving a training signal on all positions instead of 15 percent of them.

ObjectiveWhat the model learnsKnown use
Masked language modelingPredict hidden tokens from both sidesBERT, RoBERTa, ModernBERT
Next sentence predictionJudge whether two sentences are consecutiveOriginal BERT, later dropped
Replaced token detectionIdentify substituted tokensELECTRA, DeBERTa-v3
Contrastive fine-tuningPlace related texts close in vector spaceSentence-embedding models
Common pretraining objectives

Encoder-only, decoder-only and encoder-decoder designs

The three transformer families differ mainly in their attention masks and in what they output. A decoder-only model, such as GPT, uses causal attention and generates text one token at a time as an autoregressive model. An encoder-decoder model, such as T5, encodes the input bidirectionally and then decodes an output sequence with cross-attention.

The comparison below summarizes where each design is strongest. In practice, encoder-only models win on cost per prediction for fixed-label and retrieval tasks, while decoder-only models dominate open-ended generation and in-context learning.

ArchitectureAttention patternMain outputTypical tasks
Encoder-onlyBidirectionalContextual embeddingsClassification, search, tagging
Decoder-onlyCausal, left to rightNext-token probabilitiesChat, writing, code generation
Encoder-decoderBidirectional plus causalGenerated sequenceTranslation, summarization
Comparison of transformer architecture families

A frequent misconception is that encoder-only models are obsolete since the rise of LLMs. They remain the default backbone for embeddings and rerankers in retrieval pipelines, because a model with about 110 to 400 million parameters can score thousands of passages per second on modest hardware.

Practical uses and how to choose a model

Encoder-only models are used by adding a small task head and fine-tuning on labeled data, or by using their pooled vectors directly as embeddings. Fine-tuning a base-size model on a few thousand labeled examples often matches or beats prompting a far larger generative model on narrow classification tasks.

The decision table below maps common situations to a sensible configuration.

SituationRecommended choiceReason
Sentiment or topic labelsFine-tuned base model with [CLS] headHigh accuracy at low latency
Semantic searchContrastively trained bi-encoderVectors can be indexed in advance
Reranking top resultsCross-encoder over query and passageJoint attention improves precision
Long documentsLong-context encoder or chunkingStandard limit is 512 tokens
Many languagesMultilingual encoder such as XLM-RShared vocabulary across languages
Choosing an encoder-only setup

A bi-encoder encodes queries and documents separately, which allows precomputed indexes, whereas a cross-encoder reads both together and is more accurate but must run once per candidate pair. Many production systems combine them: the bi-encoder retrieves, and the cross-encoder reranks the shortlist.

Limitations and common mistakes

Encoder-only models cannot generate fluent text on their own, so they are a poor fit for chat, summarization or open answers. Their fixed context window is short in older checkpoints, and silent truncation of long inputs is one of the most common production errors.

  • Using raw [CLS] vectors as embeddings without contrastive training usually gives weak similarity scores.
  • Mismatched tokenizers between pretraining and inference corrupt inputs without raising errors.
  • Domain shift, such as legal or clinical text, often requires continued pretraining on in-domain data.
  • Overfitting during fine-tuning is common on small datasets; a learning rate near 2e-5 and two to four epochs are typical safe defaults.

Bias inherited from pretraining corpora also persists after fine-tuning, so classification outputs need auditing on representative data before deployment in sensitive decisions such as hiring or moderation.

In everyday language

An encoder-only model is a program that reads text very carefully but does not write. It looks at every word in a sentence together, in both directions, and turns the meaning into a list of numbers that a computer can compare and sort.

Think of a skilled librarian who reads a book and files it on exactly the right shelf without writing a new book. That is the job these models do: sorting, matching and labeling text quickly and at low cost, while chatbots handle the writing.

Seen in the real world

A customer support team at a mid-sized software company receives about 20,000 tickets per month and needs each one routed to billing, technical support or account security. The team fine-tunes a base-size encoder-only model on 6,000 historical tickets labeled by senior agents, using a classification head on the pooled output.

The resulting model runs on a single CPU server and labels a ticket in under 20 milliseconds. Routing accuracy reaches about 92 percent on a held-out test set, close to the agreement rate between human agents. The same encoder also produces embeddings for a search box that suggests similar solved tickets, so one compact model serves both routing and retrieval without the cost of calling a large generative model for every message.