An autoregressive model is a model that generates a sequence one element at a time, where each new element is predicted from the elements already produced. In artificial intelligence, and especially in Language models, the term describes systems such as GPT-style transformers that factorize the probability of a text into a product of conditional probabilities, one per token.
The idea comes from classical statistics, where an AR(p) process predicts a value from its previous p values. Modern autoregressive language models apply the same principle to tokens instead of numbers, and they have become the dominant architecture for text generation because the training objective is simple, scalable and directly aligned with how text is produced.
How an autoregressive model generates a sequence
An autoregressive model generates output through a loop of prediction and feedback. Given a prefix of tokens, the network computes a probability distribution over the vocabulary for the next position, a decoding strategy selects one token, and that token is appended to the input for the following step. The loop ends when the model emits an end-of-sequence token, reaches a stop sequence or hits the max tokens limit.
Mathematically, the model applies the chain rule of probability: the likelihood of a sequence x1 to xn equals the product of P(xt | x1 … xt-1) for every position t. Nothing in this decomposition is an approximation; the modeling assumptions enter only through the network that estimates each conditional term. In transformers, causal masking in self-attention enforces the rule that a position can attend only to earlier positions.
The following table breaks one generation step into its components and shows what each one contributes.
| Component | Role in the step | Typical implementation |
|---|---|---|
| Prefix encoding | Represents all tokens produced so far | Transformer layers with causal attention |
| Logits | Unnormalized scores for every vocabulary entry | Final linear projection, often 32k to 256k entries |
| Softmax with temperature | Converts scores into probabilities | Temperature between 0 and about 1.5 |
| Decoding strategy | Selects the next token | Greedy, beam search, top-k, top-p or min-p |
| KV cache | Stores past keys and values for reuse | Grows linearly with sequence length |
Training with next-token prediction and teacher forcing
An autoregressive language model is trained with next-token prediction, a self-supervised objective that minimizes the cross-entropy loss between the predicted distribution and the actual next token in a text corpus. Because every position in a document supplies its own target, a single sequence of 4,096 tokens yields thousands of training signals in one forward pass.
Training uses teacher forcing: the model always conditions on the true previous tokens, never on its own earlier guesses. Combined with causal masking, this allows all positions to be computed in parallel during training, which is the main reason decoder-only transformers scale efficiently on modern accelerators. Generation, by contrast, remains strictly sequential at inference time.
The mismatch between these two regimes is called exposure bias. During training the model never sees its own mistakes as input, so at inference a single poor token can push the context into a region the model handled rarely, and errors may compound over long outputs. Instruction tuning, reinforcement learning from feedback and careful decoding reduce, but do not remove, this effect.
Autoregressive models compared with other generative approaches
Autoregressive models differ from other generative families mainly in how they factorize the output and in what that factorization costs at inference. The table below contrasts the autoregressive approach with the closest alternatives in language modeling.
| Approach | Training objective | Generation pattern | Main strength |
|---|---|---|---|
| Autoregressive (decoder-only) | Predict the next token | Left to right, one token per step | Exact likelihood and fluent long text |
| Masked language model | Recover masked tokens from both sides | Not designed for free generation | Strong bidirectional representations |
| Encoder-decoder | Predict target tokens given a source | Autoregressive decoder over encoded input | Translation and summarization |
| Diffusion language model | Denoise corrupted token sequences | Many tokens refined in parallel | Potentially faster parallel decoding |
| Non-autoregressive model | Predict all positions at once | Single or few parallel passes | Low latency, weaker coherence |
The decisive advantage of the autoregressive approach is tractable exact likelihood, which makes evaluation by perplexity straightforward and allows logprobs to be read directly. Its main cost is sequential latency: producing 1,000 tokens requires 1,000 forward passes, even with a KV cache that avoids recomputing earlier positions.
Inference parameters that shape autoregressive output
The behavior of an autoregressive model at inference depends heavily on decoding settings, not only on its weights. The same model can produce terse, deterministic answers or varied, creative prose, depending on how the next token is chosen from the distribution.
| Parameter | Typical range | Effect on output |
|---|---|---|
| Temperature | 0 to 1.5 | Lower values sharpen the distribution and reduce variety |
| Top-p sampling | 0.8 to 0.95 | Samples only from the smallest set covering p of the mass |
| Top-k sampling | 20 to 100 | Restricts sampling to the k most likely tokens |
| Repetition penalty | 1.0 to 1.3 | Discourages tokens that already appeared |
| Max tokens | Task dependent | Caps output length and cost |
For extraction, classification and code, low temperature or greedy decoding is usually preferred because consistency matters more than variety. For brainstorming and fiction, moderate temperature with top-p keeps text varied without drifting into incoherence. Constrained decoding can further restrict choices to a grammar or JSON schema at each step.
Limitations and common misconceptions
Autoregressive generation has structural limits that practitioners should understand before relying on it. The table below links frequent problems to their underlying cause and to typical mitigations.
| Symptom | Underlying cause | Usual mitigation |
|---|---|---|
| Repetitive loops | High-probability tokens reinforce themselves | Repetition or frequency penalty, sampling |
| Confident false statements | Fluency is optimized, not factual accuracy | Retrieval grounding and verification |
| Slow long outputs | One forward pass per token | Speculative decoding, batching, smaller models |
| Early commitment to a wrong plan | No revision of emitted tokens | Chain-of-thought, reasoning tokens, self-checks |
| Degraded recall in long inputs | Uneven attention across the context window | Placing key facts near the start or end |
- Autoregressive does not mean shallow. Each step conditions on the full context, so the model can encode long-range plans in its hidden states even though it emits one token at a time.
- Autoregressive is not the same as decoder-only. Encoder-decoder systems also decode autoregressively; the term describes the factorization, not a specific architecture.
- The model cannot edit earlier tokens. Any apparent correction is new text appended after the mistake, which is why reasoning models spend extra test-time compute before answering.
The idea in brief
An autoregressive model writes the way a person types a message: one word at a time. It always looks at what is already on the screen before choosing the next word. It never sees the future. It cannot go back to erase what it wrote.
Each time it adds a piece of text, it picks the most likely continuation and repeats. This simple habit is how most chatbots produce whole answers. The model is always predicting the next piece. So the answer depends on what it learned and on how each choice is made.
A practical scenario
A support team at a mid-sized software company deploys an autoregressive language model to draft replies to customer tickets. Engineers set the temperature to 0.2 and top-p to 0.9 so that drafts stay consistent, add a stop sequence at the signature line, and cap output at 400 tokens to control cost and latency.
Early tests show that long drafts sometimes repeat the same apology. The team adds a mild repetition penalty of 1.1 and places the relevant knowledge base excerpt at the beginning of the prompt rather than in the middle. Average generation time stays near three seconds per reply, and agents report that most drafts need only light editing, which cuts handling time by roughly a third while a human still approves every message before it is sent.