An encoder-only model is a transformer architecture that keeps only the encoder stack and reads every token of an input with bidirectional self-attention, producing one contextual vector per token instead of generating new text. Among language models in artificial intelligence, it is the family behind BERT, RoBERTa, DeBERTa and many sentence-embedding models, and it is also called an encoder-only transformer.
Encoder-only models are built for understanding rather than generation. They power classification, named entity recognition, semantic search, reranking and extractive question answering, where the goal is to represent or label existing text accurately, cheaply and quickly.
How an encoder-only model processes text
An encoder-only model turns a token sequence into a sequence of contextual embeddings of the same length. Text is first split by a subword tokenizer such as WordPiece or a byte-level BPE variant, each token is mapped to a learned vector, and position information is added through absolute embeddings or rotary encodings.
The vectors then pass through a stack of identical layers, typically 12 to 24 layers in common checkpoints. Each layer combines multi-head self-attention with a position-wise feed-forward network, residual connections and layer normalization. The defining detail is the absence of a causal mask: every token can attend to every other token, both to its left and to its right, so the representation of a word reflects its full sentence context.
Because no token is ever predicted from its own future, the output is not a probability distribution over the next word. Instead, downstream components read the final hidden states. A special [CLS] token or a mean pooling step summarizes the whole input for classification and retrieval, while per-token vectors feed tagging and span extraction heads.
The following table lists the main components of a typical encoder-only model and the role each plays.
| Component | Typical setting | Role |
|---|---|---|
| Tokenizer | WordPiece or BPE, 30k to 250k entries | Splits text into subword units |
| Hidden size | 768 (base) or 1024 (large) | Width of each token vector |
| Attention heads | 12 to 16 per layer | Capture different token relations |
| Maximum sequence length | 512 tokens, newer designs 8,192 | Limits the input per pass |
| Pooling strategy | [CLS] token or mean pooling | Builds one vector per input |
Pretraining objectives for encoder-only models
Encoder-only models are pretrained with denoising objectives rather than next-token prediction, because bidirectional attention would let the model simply copy the answer. The best-known objective is masked language modeling (MLM), which hides roughly 15 percent of tokens and trains the model to recover them from the surrounding context, the approach that defines a masked language model.
Later research refined this recipe. RoBERTa showed that dropping next sentence prediction, training longer and using dynamic masking improved results. ELECTRA replaced masking with replaced token detection, in which a small generator corrupts tokens and the encoder learns to spot every substitution, giving a training signal on all positions instead of 15 percent of them.
| Objective | What the model learns | Known use |
|---|---|---|
| Masked language modeling | Predict hidden tokens from both sides | BERT, RoBERTa, ModernBERT |
| Next sentence prediction | Judge whether two sentences are consecutive | Original BERT, later dropped |
| Replaced token detection | Identify substituted tokens | ELECTRA, DeBERTa-v3 |
| Contrastive fine-tuning | Place related texts close in vector space | Sentence-embedding models |
Encoder-only, decoder-only and encoder-decoder designs
The three transformer families differ mainly in their attention masks and in what they output. A decoder-only model, such as GPT, uses causal attention and generates text one token at a time as an autoregressive model. An encoder-decoder model, such as T5, encodes the input bidirectionally and then decodes an output sequence with cross-attention.
The comparison below summarizes where each design is strongest. In practice, encoder-only models win on cost per prediction for fixed-label and retrieval tasks, while decoder-only models dominate open-ended generation and in-context learning.
| Architecture | Attention pattern | Main output | Typical tasks |
|---|---|---|---|
| Encoder-only | Bidirectional | Contextual embeddings | Classification, search, tagging |
| Decoder-only | Causal, left to right | Next-token probabilities | Chat, writing, code generation |
| Encoder-decoder | Bidirectional plus causal | Generated sequence | Translation, summarization |
A frequent misconception is that encoder-only models are obsolete since the rise of LLMs. They remain the default backbone for embeddings and rerankers in retrieval pipelines, because a model with about 110 to 400 million parameters can score thousands of passages per second on modest hardware.
Practical uses and how to choose a model
Encoder-only models are used by adding a small task head and fine-tuning on labeled data, or by using their pooled vectors directly as embeddings. Fine-tuning a base-size model on a few thousand labeled examples often matches or beats prompting a far larger generative model on narrow classification tasks.
The decision table below maps common situations to a sensible configuration.
| Situation | Recommended choice | Reason |
|---|---|---|
| Sentiment or topic labels | Fine-tuned base model with [CLS] head | High accuracy at low latency |
| Semantic search | Contrastively trained bi-encoder | Vectors can be indexed in advance |
| Reranking top results | Cross-encoder over query and passage | Joint attention improves precision |
| Long documents | Long-context encoder or chunking | Standard limit is 512 tokens |
| Many languages | Multilingual encoder such as XLM-R | Shared vocabulary across languages |
A bi-encoder encodes queries and documents separately, which allows precomputed indexes, whereas a cross-encoder reads both together and is more accurate but must run once per candidate pair. Many production systems combine them: the bi-encoder retrieves, and the cross-encoder reranks the shortlist.
Limitations and common mistakes
Encoder-only models cannot generate fluent text on their own, so they are a poor fit for chat, summarization or open answers. Their fixed context window is short in older checkpoints, and silent truncation of long inputs is one of the most common production errors.
- Using raw [CLS] vectors as embeddings without contrastive training usually gives weak similarity scores.
- Mismatched tokenizers between pretraining and inference corrupt inputs without raising errors.
- Domain shift, such as legal or clinical text, often requires continued pretraining on in-domain data.
- Overfitting during fine-tuning is common on small datasets; a learning rate near 2e-5 and two to four epochs are typical safe defaults.
Bias inherited from pretraining corpora also persists after fine-tuning, so classification outputs need auditing on representative data before deployment in sensitive decisions such as hiring or moderation.
In everyday language
An encoder-only model is a program that reads text very carefully but does not write. It looks at every word in a sentence together, in both directions, and turns the meaning into a list of numbers that a computer can compare and sort.
Think of a skilled librarian who reads a book and files it on exactly the right shelf without writing a new book. That is the job these models do: sorting, matching and labeling text quickly and at low cost, while chatbots handle the writing.
Seen in the real world
A customer support team at a mid-sized software company receives about 20,000 tickets per month and needs each one routed to billing, technical support or account security. The team fine-tunes a base-size encoder-only model on 6,000 historical tickets labeled by senior agents, using a classification head on the pooled output.
The resulting model runs on a single CPU server and labels a ticket in under 20 milliseconds. Routing accuracy reaches about 92 percent on a held-out test set, close to the agreement rate between human agents. The same encoder also produces embeddings for a search box that suggests similar solved tickets, so one compact model serves both routing and retrieval without the cost of calling a large generative model for every message.