Top-p sampling

Also known as: nucleus sampling

On this page 7

Top-p sampling, also called nucleus sampling, is a decoding method that draws each next token from the smallest set of candidate tokens whose cumulative probability reaches a threshold p, then renormalizes those probabilities and samples among them. In language models, it is one of the most widely used controls for balancing coherence against diversity during text generation.

The method was introduced to fix a weakness of fixed-size truncation: the number of sensible continuations changes from one position to the next. By defining the candidate pool through probability mass instead of a fixed count, top-p sampling adapts automatically, keeping the pool narrow when the model is confident and widening it when many continuations are plausible.

How top-p sampling selects the next token

Top-p sampling works on the probability distribution that an autoregressive model produces at every generation step. The model outputs logits for each vocabulary entry, a softmax converts them into probabilities, and the sampler then truncates the long tail of unlikely tokens before making a random draw.

The procedure is short and deterministic up to the final draw. Because the cutoff depends on the shape of the distribution, the size of the nucleus varies at every step, from a single token to several hundred.

  1. Compute probabilities from the logits, usually after temperature scaling has been applied.
  2. Sort all tokens by probability in descending order.
  3. Add probabilities from the top until the running sum reaches or exceeds p.
  4. Discard every token outside this nucleus and renormalize the remaining probabilities so they sum to 1.
  5. Draw one token at random from the renormalized nucleus.

The table below shows how the same threshold produces nuclei of very different sizes depending on how confident the model is at a given position.

Generation contextTop token probabilityTypical nucleus sizeEffect
Completing a fixed idiom0.951 tokenBehaves like greedy decoding
Choosing the next word in a technical sentence0.553 to 8 tokensSmall, safe variation
Starting a new sentence in a story0.08Dozens to hundreds of tokensBroad creative choice
Nucleus size at p = 0.9 for different distribution shapes

Typical values of the p parameter

The parameter p ranges from 0 to 1, and most production systems use values between 0.8 and 0.95. A value of 1.0 disables truncation entirely, while values near 0 collapse the nucleus to the single most likely token, which is functionally equivalent to greedy decoding.

The right value depends on the task, the model and whether temperature is also being changed. The following ranges are common starting points rather than fixed rules.

Value of pBehaviorTypical use
0.1 to 0.5Very conservative, near-deterministicExtraction, classification, structured output
0.7 to 0.9Focused with moderate varietyAssistants, summaries, technical answers
0.9 to 0.95Natural, varied phrasingGeneral chat, drafting, dialogue
0.95 to 1.0Wide exploration, higher risk of driftBrainstorming, fiction, idea generation
Common p values and their effects

Small changes at the high end matter more than they appear. Moving from 0.95 to 0.99 can admit a large number of low-probability tokens in flat distributions, which is where incoherent or off-topic words tend to enter the output.

Top-p compared with other decoding strategies

Top-p sampling is one member of a family of truncation samplers, and each defines the candidate pool differently. Its main advantage over top-k sampling is adaptivity to distribution shape; its main weakness, compared with newer methods such as min-p sampling, is that it can still admit many weak tokens when the distribution is flat.

The comparison below summarizes how the most common decoding strategies decide which tokens remain eligible.

StrategyCandidate ruleStrengthWeakness
Greedy decodingAlways the single top tokenFast and reproducibleRepetitive, often bland
Beam searchKeeps several best partial sequencesGood for translationGeneric text in open-ended tasks
Top-k samplingFixed number k of top tokensSimple and predictable costIgnores distribution shape
Top-p samplingSmallest set reaching mass pAdapts to model confidenceFlat distributions admit weak tokens
Min-p samplingTokens above a fraction of the top probabilityScales with confidenceLess established in older toolkits
How decoding strategies choose candidate tokens

Many inference engines allow these filters to be combined in sequence, for example top-k first to cap computation, then top-p to trim the tail. When several filters are active, the most restrictive one determines the effective pool at each step.

Interaction with temperature and other settings

Temperature and top-p both influence randomness, but they act at different points. Temperature reshapes the distribution by dividing the logits before the softmax, while top-p truncates the reshaped distribution afterward. In most implementations, temperature is applied first, so a higher temperature flattens the curve and therefore enlarges the nucleus for the same p.

Because the two settings compound, many provider guides recommend adjusting one of them at a time. Raising both together makes it difficult to tell which setting caused a change in output quality.

GoalSuggested settingReason
Reproducible test outputsTemperature 0 or p near 0Removes sampling variance
Reliable factual answersp 0.8 to 0.9, moderate temperatureTrims rare, error-prone tokens
Creative writingp 0.95, temperature 0.8 to 1.0Allows varied word choice
Strict JSON or codeLow p plus constrained decodingFormat validity matters more than variety
Practical configuration choices

Other generation controls remain independent. A repetition penalty or frequency penalty modifies the logits before sampling, and a stop sequence or max tokens limit ends generation regardless of how the nucleus was formed.

Limitations and common misconceptions

Top-p sampling controls how output is sampled, not what the model knows. It cannot remove hallucination at its source, because a confidently wrong token sits inside the nucleus just as a confidently correct one does.

  • Low p does not guarantee accuracy. It only makes the model repeat its own most likely belief more consistently.
  • High p is not the same as high temperature. Top-p never raises the probability of rare tokens; it only decides whether they stay eligible.
  • Flat distributions remain a risk. When the model is uncertain, a p of 0.95 can admit hundreds of weak tokens and produce drift in long generations.
  • Implementations differ slightly. Some libraries include the token that crosses the threshold and some exclude it, and the order of filters is not standardized.
  • Results are not reproducible by default. Identical settings still give different outputs unless a fixed random seed is used.

Put simply

When a language model writes, it guesses the next word over and over. For each guess it has a ranked list of candidates with odds attached. Top-p sampling tells the model to keep only the favorites that together cover most of the odds, for example 90 percent, and to pick randomly among them.

Think of ordering at a restaurant. If one dish is clearly the best, you choose it almost every time. If five dishes are equally good, you rotate among them. The method behaves the same way, so the text stays sensible while still varying from one answer to the next.

In practice

A software team builds a customer support assistant on a hosted language model. In early tests with p set to 1.0 and temperature at 1.0, the assistant sometimes invents unusual product names and wanders off topic in longer replies. The team reviews 200 transcripts and finds that most errors begin with a single improbable token that the next sentences then build upon.

The engineers lower p to 0.85 and keep temperature at 0.7. Replies still vary in wording, so customers do not see identical canned text, but the rare tokens that started the errors are now excluded. For the order-lookup feature, which must return strict JSON, they set p to 0.1 and add constrained decoding. In the next review cycle, off-topic replies drop noticeably and malformed JSON responses disappear, while satisfaction ratings for conversational answers stay level.