Next-token prediction is the task of estimating a probability distribution over the vocabulary for the token that follows a given sequence, and it is the core training objective and generation mechanism of modern language models in artificial intelligence. A model sees a prefix such as “The capital of France is”, assigns a score to every possible next token, and learns to put high probability on the token that actually followed in its training text.
The same objective serves two roles. During pretraining, it turns raw text into billions of supervised examples without human labels. During inference, the model applies it repeatedly, appending one chosen token at a time, which is why decoder-only systems such as GPT are described as autoregressive models.
How next-token prediction works inside a model
Next-token prediction works by mapping a token sequence to a vector of logits, one per vocabulary entry, and converting those logits into probabilities with the softmax function. Text is first split by a tokenizer, often based on byte-pair encoding, into integer IDs. A transformer with causal attention masking then processes the sequence so that each position can attend only to earlier positions, never to the future.
At the final layer, the hidden state of each position is projected onto the vocabulary, which usually contains 32,000 to more than 200,000 tokens. Because every position predicts its own successor, a single forward pass over a training sequence of length n yields n prediction targets at once. This parallelism, called teacher forcing, is a major reason the objective scales so efficiently on modern accelerators.
The table below traces the main stages from input text to a predicted token.
| Stage | What happens | Output |
|---|---|---|
| Tokenization | Text is split into subword units and mapped to IDs | Sequence of integer token IDs |
| Embedding | Each ID is turned into a dense vector with position information | Matrix of input vectors |
| Causal transformer layers | Attention mixes information from earlier positions only | Contextual hidden states |
| Output projection | The last hidden state is multiplied by the unembedding matrix | One logit per vocabulary token |
| Softmax and selection | Logits become probabilities, then a token is chosen | The next token |
The training objective and its loss
The training signal for next-token prediction is the cross-entropy loss, the negative log probability the model assigned to the true next token, averaged over all positions. Minimizing it is equivalent to maximum likelihood estimation of the training text under a factorized model, in which the probability of a sequence equals the product of the conditional probabilities of its tokens.
Loss values are usually reported in nats or bits per token, and their exponential gives perplexity, which can be read as the effective number of equally likely choices the model faces at each step. Empirical scaling laws show that loss decreases smoothly and predictably as parameters, data and compute grow, although the exact exponents depend on the data mix and architecture.
The following table summarizes the quantities practitioners monitor when training with this objective.
| Quantity | Definition | Typical interpretation |
|---|---|---|
| Cross-entropy loss | Mean negative log probability of the true token | Lower values mean better prediction |
| Perplexity | Exponential of the per-token loss | Effective branching factor per step |
| Bits per byte | Loss normalized by raw text length | Compares models with different tokenizers |
| Token accuracy | Share of positions where the top token is correct | Coarse; ignores calibration |
A subtle point is that perplexity is tokenizer-dependent. Two models with different vocabularies cannot be compared on perplexity directly, which is why research reports often use bits per byte instead.
Next-token prediction compared with other language modeling objectives
Next-token prediction differs from other objectives mainly in which context the model may see and what it must reconstruct. Encoder-only and masked language models such as BERT predict hidden tokens using context on both sides, which suits classification and retrieval but does not produce a natural generation procedure. Diffusion language models refine many tokens in parallel over several denoising steps.
| Objective | Context used | Natural strength | Generation style |
|---|---|---|---|
| Next-token prediction | Left context only | Open-ended text generation | One token at a time |
| Masked language modeling | Left and right context | Understanding and embeddings | Not directly generative |
| Span corruption | Context around masked spans | Sequence-to-sequence tasks | Encoder-decoder output |
| Diffusion objectives | Noisy full sequence | Parallel refinement | Several tokens per step |
| Multi-token prediction | Left context only | Faster decoding and planning signal | Several future tokens |
Some recent models add auxiliary multi-token heads that predict two or more future tokens during training. These heads are typically used for speculative decoding at inference, while the primary objective remains next-token prediction.
From probabilities to generated text
Generated text emerges from next-token prediction through a decoding strategy that turns each probability distribution into one concrete token. The chosen token is appended to the context, and the loop repeats until a stop sequence, an end-of-sequence token or a max tokens limit ends it. A key-value cache stores intermediate attention results so that each new step does not recompute the entire prefix.
Decoding settings strongly shape output quality even when the model weights are fixed. The table lists common choices and their effects.
| Method | How it selects a token | Typical effect |
|---|---|---|
| Greedy decoding | Always picks the most probable token | Deterministic, can loop or repeat |
| Temperature sampling | Rescales logits before sampling | Higher values add diversity and risk |
| Top-k sampling | Samples from the k most likely tokens | Removes the long unlikely tail |
| Top-p sampling | Samples from the smallest set covering mass p | Adapts candidate count to confidence |
| Beam search | Keeps several partial sequences in parallel | Good for translation, bland for chat |
Because every token is conditioned on earlier choices, errors compound: an early unlikely token can steer the rest of the output in an unintended direction. This effect, sometimes called exposure bias, arises because training always uses the true prefix while generation uses the model’s own output.
Capabilities, limitations and common misconceptions
Predicting text well across a broad corpus forces a model to capture grammar, facts, reasoning patterns and style, because all of them reduce prediction error. This explains why in-context learning and other emergent abilities appear in large base models trained on nothing but next-token prediction. Instruction tuning and preference optimization later reshape the same distribution toward helpful behavior.
The objective also has well-documented limitations that experienced practitioners account for:
- Fluency is rewarded over truth. A plausible continuation can score well even when it is false, which contributes to hallucination.
- Knowledge is frozen at the knowledge cutoff of the training data unless retrieval supplies fresh context.
- Left-to-right generation makes some tasks harder, such as satisfying constraints that depend on later text.
- Probabilities are not certainty. Logprobs from tuned chat models are often less calibrated than those of the base model.
A frequent misconception holds that next-token prediction implies no internal planning. Interpretability studies indicate that models can represent information about tokens several positions ahead, so the objective is local but the learned computation need not be. Whether this amounts to genuine understanding remains an open and disputed research question.
In everyday language
Next-token prediction is a very advanced version of finishing someone else’s sentence. The model reads the words so far and guesses which small piece of text should come next, then adds that piece and guesses again. Every answer is built one small piece at a time.
To get good at this guessing, the model practices on enormous amounts of written text, checking its guess against what really came next. Over time, good guessing requires learning grammar, facts and patterns of reasoning. The method is powerful, but it rewards answers that sound right, so a confident answer is not always a correct one.
Seen in the real world
A software team builds an internal assistant that drafts replies to customer support tickets using an open-weight model. When an agent opens a ticket about a failed payment, the assistant receives the ticket text and a short system prompt, then generates a reply one token at a time with a temperature of 0.3 and top-p of 0.9, stopping at a limit of 400 tokens.
During testing, the team logs the logprobs of each generated token. They notice that refund amounts and policy dates often receive low probabilities, a sign that the model is guessing. They therefore insert the customer’s actual order data and the current refund policy into the context before generation. Low-confidence spans drop sharply, and reviewers accept about 70 percent of drafts with minor edits, compared with fewer than half before the change.