Autoregressive model

Also known as: autoregressive language model

On this page 7

An autoregressive model is a model that generates a sequence one element at a time, where each new element is predicted from the elements already produced. In artificial intelligence, and especially in Language models, the term describes systems such as GPT-style transformers that factorize the probability of a text into a product of conditional probabilities, one per token.

The idea comes from classical statistics, where an AR(p) process predicts a value from its previous p values. Modern autoregressive language models apply the same principle to tokens instead of numbers, and they have become the dominant architecture for text generation because the training objective is simple, scalable and directly aligned with how text is produced.

How an autoregressive model generates a sequence

An autoregressive model generates output through a loop of prediction and feedback. Given a prefix of tokens, the network computes a probability distribution over the vocabulary for the next position, a decoding strategy selects one token, and that token is appended to the input for the following step. The loop ends when the model emits an end-of-sequence token, reaches a stop sequence or hits the max tokens limit.

Mathematically, the model applies the chain rule of probability: the likelihood of a sequence x1 to xn equals the product of P(xt | x1 … xt-1) for every position t. Nothing in this decomposition is an approximation; the modeling assumptions enter only through the network that estimates each conditional term. In transformers, causal masking in self-attention enforces the rule that a position can attend only to earlier positions.

The following table breaks one generation step into its components and shows what each one contributes.

ComponentRole in the stepTypical implementation
Prefix encodingRepresents all tokens produced so farTransformer layers with causal attention
LogitsUnnormalized scores for every vocabulary entryFinal linear projection, often 32k to 256k entries
Softmax with temperatureConverts scores into probabilitiesTemperature between 0 and about 1.5
Decoding strategySelects the next tokenGreedy, beam search, top-k, top-p or min-p
KV cacheStores past keys and values for reuseGrows linearly with sequence length
Components of a single autoregressive generation step

Training with next-token prediction and teacher forcing

An autoregressive language model is trained with next-token prediction, a self-supervised objective that minimizes the cross-entropy loss between the predicted distribution and the actual next token in a text corpus. Because every position in a document supplies its own target, a single sequence of 4,096 tokens yields thousands of training signals in one forward pass.

Training uses teacher forcing: the model always conditions on the true previous tokens, never on its own earlier guesses. Combined with causal masking, this allows all positions to be computed in parallel during training, which is the main reason decoder-only transformers scale efficiently on modern accelerators. Generation, by contrast, remains strictly sequential at inference time.

The mismatch between these two regimes is called exposure bias. During training the model never sees its own mistakes as input, so at inference a single poor token can push the context into a region the model handled rarely, and errors may compound over long outputs. Instruction tuning, reinforcement learning from feedback and careful decoding reduce, but do not remove, this effect.

Autoregressive models compared with other generative approaches

Autoregressive models differ from other generative families mainly in how they factorize the output and in what that factorization costs at inference. The table below contrasts the autoregressive approach with the closest alternatives in language modeling.

ApproachTraining objectiveGeneration patternMain strength
Autoregressive (decoder-only)Predict the next tokenLeft to right, one token per stepExact likelihood and fluent long text
Masked language modelRecover masked tokens from both sidesNot designed for free generationStrong bidirectional representations
Encoder-decoderPredict target tokens given a sourceAutoregressive decoder over encoded inputTranslation and summarization
Diffusion language modelDenoise corrupted token sequencesMany tokens refined in parallelPotentially faster parallel decoding
Non-autoregressive modelPredict all positions at onceSingle or few parallel passesLow latency, weaker coherence
Autoregressive modeling versus related approaches

The decisive advantage of the autoregressive approach is tractable exact likelihood, which makes evaluation by perplexity straightforward and allows logprobs to be read directly. Its main cost is sequential latency: producing 1,000 tokens requires 1,000 forward passes, even with a KV cache that avoids recomputing earlier positions.

Inference parameters that shape autoregressive output

The behavior of an autoregressive model at inference depends heavily on decoding settings, not only on its weights. The same model can produce terse, deterministic answers or varied, creative prose, depending on how the next token is chosen from the distribution.

ParameterTypical rangeEffect on output
Temperature0 to 1.5Lower values sharpen the distribution and reduce variety
Top-p sampling0.8 to 0.95Samples only from the smallest set covering p of the mass
Top-k sampling20 to 100Restricts sampling to the k most likely tokens
Repetition penalty1.0 to 1.3Discourages tokens that already appeared
Max tokensTask dependentCaps output length and cost
Common decoding parameters and their effects

For extraction, classification and code, low temperature or greedy decoding is usually preferred because consistency matters more than variety. For brainstorming and fiction, moderate temperature with top-p keeps text varied without drifting into incoherence. Constrained decoding can further restrict choices to a grammar or JSON schema at each step.

Limitations and common misconceptions

Autoregressive generation has structural limits that practitioners should understand before relying on it. The table below links frequent problems to their underlying cause and to typical mitigations.

SymptomUnderlying causeUsual mitigation
Repetitive loopsHigh-probability tokens reinforce themselvesRepetition or frequency penalty, sampling
Confident false statementsFluency is optimized, not factual accuracyRetrieval grounding and verification
Slow long outputsOne forward pass per tokenSpeculative decoding, batching, smaller models
Early commitment to a wrong planNo revision of emitted tokensChain-of-thought, reasoning tokens, self-checks
Degraded recall in long inputsUneven attention across the context windowPlacing key facts near the start or end
Typical problems of autoregressive generation
  • Autoregressive does not mean shallow. Each step conditions on the full context, so the model can encode long-range plans in its hidden states even though it emits one token at a time.
  • Autoregressive is not the same as decoder-only. Encoder-decoder systems also decode autoregressively; the term describes the factorization, not a specific architecture.
  • The model cannot edit earlier tokens. Any apparent correction is new text appended after the mistake, which is why reasoning models spend extra test-time compute before answering.

The idea in brief

An autoregressive model writes the way a person types a message: one word at a time. It always looks at what is already on the screen before choosing the next word. It never sees the future. It cannot go back to erase what it wrote.

Each time it adds a piece of text, it picks the most likely continuation and repeats. This simple habit is how most chatbots produce whole answers. The model is always predicting the next piece. So the answer depends on what it learned and on how each choice is made.

A practical scenario

A support team at a mid-sized software company deploys an autoregressive language model to draft replies to customer tickets. Engineers set the temperature to 0.2 and top-p to 0.9 so that drafts stay consistent, add a stop sequence at the signature line, and cap output at 400 tokens to control cost and latency.

Early tests show that long drafts sometimes repeat the same apology. The team adds a mild repetition penalty of 1.1 and places the relevant knowledge base excerpt at the beginning of the prompt rather than in the middle. Average generation time stays near three seconds per reply, and agents report that most drafts need only light editing, which cuts handling time by roughly a third while a human still approves every message before it is sent.