Decoder-only model

Also known as: decoder-only transformer

On this page 7

A decoder-only model is a transformer architecture that keeps only the decoder stack and uses causal self-attention, so every position can attend to earlier tokens but never to later ones. Trained on next-token prediction, it reads a sequence from left to right and generates text one token at a time, feeding each output back as new input.

Language models built this way, often called decoder-only transformers, dominate current work in artificial intelligence: the GPT family, Llama, Mistral, Claude and most open-weight chat models follow this design. Its appeal lies in one simple training objective that scales well with data and compute and that turns naturally into open-ended generation.

How a decoder-only model processes text

A decoder-only model converts text into token embeddings, adds position information, and passes the result through a stack of identical transformer blocks before a final projection produces logits over the vocabulary. Each block contains masked multi-head self-attention, a feed-forward network, residual connections and normalization layers.

The defining element is the causal attention mask. It sets attention scores for all future positions to negative infinity before the softmax, so token 50 can draw information from tokens 1 to 49 and itself, but not from token 51. This restriction makes the model autoregressive: the probability of a whole sequence factorizes into a product of conditional probabilities, one for each next token.

The following table lists the main components and what each contributes to the computation.

ComponentFunctionTypical choice in recent models
Token embeddingMaps token IDs to vectorsTied or untied with output layer
Positional encodingInjects token orderRotary position embedding (RoPE)
Causal self-attentionMixes information from earlier tokensGrouped-query attention
Feed-forward networkTransforms each position independentlySwiGLU activation
NormalizationStabilizes training of deep stacksPre-norm RMSNorm
Output projectionProduces logits for every vocabulary entrySoftmax over 32,000 to 256,000 tokens
Core components of a decoder-only transformer

Training and inference in decoder-only language models

Pretraining uses teacher forcing: the model sees a full sequence, predicts every next token in parallel thanks to the mask, and minimizes cross-entropy loss across all positions. A single document of 4,000 tokens therefore yields roughly 4,000 training signals, which makes the objective very data-efficient compared with objectives that score only a few masked positions.

Inference has two distinct phases. During prefill, the model processes the whole prompt in one parallel pass and stores keys and values for every layer in the KV cache. During decoding, it generates one token per step, reusing the cache so each new token attends to stored history instead of recomputing it. Prefill is usually compute-bound, whereas decoding is usually memory-bandwidth-bound.

The output of each decoding step is a probability distribution, and a decoding strategy such as greedy decoding, top-p sampling or beam search turns it into an actual token. Settings like temperature, stop sequences and max tokens act at this stage and do not change the network itself.

The table below summarizes how the two inference phases differ in cost and bottleneck.

PropertyPrefillDecoding
Tokens processed per passEntire prompt at onceOne new token per step
Main bottleneckArithmetic throughputMemory bandwidth
Latency metricTime to first tokenTime per output token
KV cache roleWrittenRead and extended
Optimization examplesChunked prefill, FlashAttentionSpeculative decoding, batching
Prefill and decoding compared

Decoder-only, encoder-only and encoder-decoder architectures

The original transformer of 2017 paired an encoder with a decoder for machine translation, and later research split the design into three families. An encoder-only model such as BERT uses bidirectional attention and a masked language model objective, which suits classification and embeddings but not free generation. An encoder-decoder model such as T5 encodes the input bidirectionally and then generates output with a separate decoder linked by cross-attention.

Decoder-only designs won the race for general assistants largely because of simplicity at scale. One stack, one objective and one token stream handle both input and output, so any task can be framed as a continuation of a prompt. This uniformity enables in-context learning and few-shot prompting without task-specific heads.

ArchitectureAttention patternTraining objectiveTypical strengths
Decoder-onlyCausal, left to rightNext-token predictionOpen-ended generation, chat, code
Encoder-onlyBidirectionalMasked token predictionClassification, search embeddings
Encoder-decoderBidirectional encoder, causal decoderSpan corruption or sequence-to-sequenceTranslation, summarization of fixed inputs
Comparison of the three transformer families

A common misconception holds that decoder-only models cannot understand context because they lack bidirectional attention. In practice, the final prompt token attends to the entire prompt, and large models build rich representations of the input; the restriction mainly affects how embeddings for individual tokens are formed, which is why encoder-only models still dominate some retrieval tasks.

Variants and design choices

Decoder-only models share one skeleton but differ in parameter count, attention efficiency and sparsity. These choices determine how much memory a model needs, how long a context window it supports and how fast it answers.

VariantWhat changesMain benefitTrade-off
Dense modelAll parameters active per tokenPredictable qualityHighest compute per token
Mixture of expertsRouter activates a few expert networksMore capacity per unit of computeLarge memory footprint, routing instability
Grouped-query attentionSeveral query heads share key-value headsSmaller KV cacheSlight quality cost if overly aggressive
Sliding-window attentionEach layer sees a limited spanCheaper long sequencesWeaker direct long-range recall
Prefix language modelBidirectional attention over the prompt onlyBetter input encodingLess uniform training and caching
Common decoder-only design variants

Size categories are fluid. A small language model may have 1 to 8 billion parameters and run on a laptop, while frontier models are believed to use hundreds of billions of active or total parameters; exact figures for closed models are generally not disclosed.

Limitations of the decoder-only approach

The left-to-right structure creates specific failure modes that practitioners should anticipate. Because generation commits to each token before seeing what follows, an early mistake propagates through the rest of the answer, which contributes to hallucination and inconsistent reasoning.

  • Sequential decoding latency: output length directly drives response time, since tokens cannot be generated fully in parallel.
  • Quadratic attention cost: standard attention grows with the square of sequence length, which makes long context expensive.
  • KV cache memory: at long contexts and large batch sizes the cache can exceed the size of the model weights.
  • Lost in the middle: retrieval of facts placed mid-prompt is often weaker than at the beginning or end.
  • Reversal weakness: models trained on statements of the form A is B often fail to answer the inverse question reliably.

Mitigations include speculative decoding, quantization of weights and cache, retrieval augmentation and reasoning techniques such as chain-of-thought. Diffusion language models and other non-autoregressive approaches are being explored as alternatives, but as of now they have not displaced decoder-only systems for general-purpose use.

The simple explanation

A decoder-only model writes the way a person types a message with a very good autocomplete. It looks at everything written so far, guesses the most fitting next word piece, adds it, and repeats. It never peeks ahead, because the future part does not exist yet.

Training is just as simple. The model reads enormous amounts of text and, at every point, practices guessing what comes next. After enough practice, guessing the next piece well requires knowing grammar, facts and patterns of reasoning, which is why one plain goal produces such capable chat assistants.

How it is used in practice

A software company deploys an 8-billion-parameter decoder-only model on a single GPU with 24 GB of memory to answer internal support questions. Engineers quantize the weights to 4 bits, enable grouped-query attention to shrink the KV cache, and cap the context window at 16,000 tokens so that eight conversations can be served at once.

Monitoring shows that prefill of a typical 3,000-token prompt with retrieved documentation takes about 0.3 seconds, while generating a 400-token answer takes about 6 seconds, confirming that decoding speed dominates user wait time. Adding speculative decoding with a small draft model cuts the answer time to roughly 3.5 seconds without changing output quality, and the team keeps that configuration in production.