Next-token prediction

On this page 7

Next-token prediction is the task of estimating a probability distribution over the vocabulary for the token that follows a given sequence, and it is the core training objective and generation mechanism of modern language models in artificial intelligence. A model sees a prefix such as “The capital of France is”, assigns a score to every possible next token, and learns to put high probability on the token that actually followed in its training text.

The same objective serves two roles. During pretraining, it turns raw text into billions of supervised examples without human labels. During inference, the model applies it repeatedly, appending one chosen token at a time, which is why decoder-only systems such as GPT are described as autoregressive models.

How next-token prediction works inside a model

Next-token prediction works by mapping a token sequence to a vector of logits, one per vocabulary entry, and converting those logits into probabilities with the softmax function. Text is first split by a tokenizer, often based on byte-pair encoding, into integer IDs. A transformer with causal attention masking then processes the sequence so that each position can attend only to earlier positions, never to the future.

At the final layer, the hidden state of each position is projected onto the vocabulary, which usually contains 32,000 to more than 200,000 tokens. Because every position predicts its own successor, a single forward pass over a training sequence of length n yields n prediction targets at once. This parallelism, called teacher forcing, is a major reason the objective scales so efficiently on modern accelerators.

The table below traces the main stages from input text to a predicted token.

StageWhat happensOutput
TokenizationText is split into subword units and mapped to IDsSequence of integer token IDs
EmbeddingEach ID is turned into a dense vector with position informationMatrix of input vectors
Causal transformer layersAttention mixes information from earlier positions onlyContextual hidden states
Output projectionThe last hidden state is multiplied by the unembedding matrixOne logit per vocabulary token
Softmax and selectionLogits become probabilities, then a token is chosenThe next token
Stages of a single next-token prediction step

The training objective and its loss

The training signal for next-token prediction is the cross-entropy loss, the negative log probability the model assigned to the true next token, averaged over all positions. Minimizing it is equivalent to maximum likelihood estimation of the training text under a factorized model, in which the probability of a sequence equals the product of the conditional probabilities of its tokens.

Loss values are usually reported in nats or bits per token, and their exponential gives perplexity, which can be read as the effective number of equally likely choices the model faces at each step. Empirical scaling laws show that loss decreases smoothly and predictably as parameters, data and compute grow, although the exact exponents depend on the data mix and architecture.

The following table summarizes the quantities practitioners monitor when training with this objective.

QuantityDefinitionTypical interpretation
Cross-entropy lossMean negative log probability of the true tokenLower values mean better prediction
PerplexityExponential of the per-token lossEffective branching factor per step
Bits per byteLoss normalized by raw text lengthCompares models with different tokenizers
Token accuracyShare of positions where the top token is correctCoarse; ignores calibration
Key quantities in next-token training

A subtle point is that perplexity is tokenizer-dependent. Two models with different vocabularies cannot be compared on perplexity directly, which is why research reports often use bits per byte instead.

Next-token prediction compared with other language modeling objectives

Next-token prediction differs from other objectives mainly in which context the model may see and what it must reconstruct. Encoder-only and masked language models such as BERT predict hidden tokens using context on both sides, which suits classification and retrieval but does not produce a natural generation procedure. Diffusion language models refine many tokens in parallel over several denoising steps.

ObjectiveContext usedNatural strengthGeneration style
Next-token predictionLeft context onlyOpen-ended text generationOne token at a time
Masked language modelingLeft and right contextUnderstanding and embeddingsNot directly generative
Span corruptionContext around masked spansSequence-to-sequence tasksEncoder-decoder output
Diffusion objectivesNoisy full sequenceParallel refinementSeveral tokens per step
Multi-token predictionLeft context onlyFaster decoding and planning signalSeveral future tokens
Comparison of language modeling objectives

Some recent models add auxiliary multi-token heads that predict two or more future tokens during training. These heads are typically used for speculative decoding at inference, while the primary objective remains next-token prediction.

From probabilities to generated text

Generated text emerges from next-token prediction through a decoding strategy that turns each probability distribution into one concrete token. The chosen token is appended to the context, and the loop repeats until a stop sequence, an end-of-sequence token or a max tokens limit ends it. A key-value cache stores intermediate attention results so that each new step does not recompute the entire prefix.

Decoding settings strongly shape output quality even when the model weights are fixed. The table lists common choices and their effects.

MethodHow it selects a tokenTypical effect
Greedy decodingAlways picks the most probable tokenDeterministic, can loop or repeat
Temperature samplingRescales logits before samplingHigher values add diversity and risk
Top-k samplingSamples from the k most likely tokensRemoves the long unlikely tail
Top-p samplingSamples from the smallest set covering mass pAdapts candidate count to confidence
Beam searchKeeps several partial sequences in parallelGood for translation, bland for chat
Decoding choices applied to next-token distributions

Because every token is conditioned on earlier choices, errors compound: an early unlikely token can steer the rest of the output in an unintended direction. This effect, sometimes called exposure bias, arises because training always uses the true prefix while generation uses the model’s own output.

Capabilities, limitations and common misconceptions

Predicting text well across a broad corpus forces a model to capture grammar, facts, reasoning patterns and style, because all of them reduce prediction error. This explains why in-context learning and other emergent abilities appear in large base models trained on nothing but next-token prediction. Instruction tuning and preference optimization later reshape the same distribution toward helpful behavior.

The objective also has well-documented limitations that experienced practitioners account for:

  • Fluency is rewarded over truth. A plausible continuation can score well even when it is false, which contributes to hallucination.
  • Knowledge is frozen at the knowledge cutoff of the training data unless retrieval supplies fresh context.
  • Left-to-right generation makes some tasks harder, such as satisfying constraints that depend on later text.
  • Probabilities are not certainty. Logprobs from tuned chat models are often less calibrated than those of the base model.

A frequent misconception holds that next-token prediction implies no internal planning. Interpretability studies indicate that models can represent information about tokens several positions ahead, so the objective is local but the learned computation need not be. Whether this amounts to genuine understanding remains an open and disputed research question.

In everyday language

Next-token prediction is a very advanced version of finishing someone else’s sentence. The model reads the words so far and guesses which small piece of text should come next, then adds that piece and guesses again. Every answer is built one small piece at a time.

To get good at this guessing, the model practices on enormous amounts of written text, checking its guess against what really came next. Over time, good guessing requires learning grammar, facts and patterns of reasoning. The method is powerful, but it rewards answers that sound right, so a confident answer is not always a correct one.

Seen in the real world

A software team builds an internal assistant that drafts replies to customer support tickets using an open-weight model. When an agent opens a ticket about a failed payment, the assistant receives the ticket text and a short system prompt, then generates a reply one token at a time with a temperature of 0.3 and top-p of 0.9, stopping at a limit of 400 tokens.

During testing, the team logs the logprobs of each generated token. They notice that refund amounts and policy dates often receive low probabilities, a sign that the model is guessing. They therefore insert the customer’s actual order data and the current refund policy into the context before generation. Low-confidence spans drop sharply, and reviewers accept about 70 percent of drafts with minor edits, compared with fewer than half before the change.