Top-k sampling

On this page 7

Top-k sampling is a decoding strategy for language models in which the model, at every generation step, keeps only the k tokens with the highest probability, renormalizes their probabilities so they sum to one, and draws the next token at random from that reduced set. It sits between greedy decoding, which always takes the single best token, and pure sampling from the full vocabulary, which can occasionally select absurd low-probability tokens.

The method became widely known through early neural story generation and the GPT-2 release, where values such as k = 40 produced text that was varied yet coherent. In modern large language models it is usually combined with temperature and often with top-p or min-p sampling, and it remains a standard parameter in inference APIs and open-source serving frameworks.

How top-k sampling filters the next-token distribution

Top-k sampling works by truncating the probability distribution that the model produces over its vocabulary before a token is drawn. An autoregressive model outputs one logit per vocabulary entry, often 32,000 to more than 200,000 values, and a softmax turns these logits into probabilities.

The procedure at each step is short and deterministic up to the final random draw. Because it only needs a partial sort of the logits, it adds negligible latency compared with the forward pass of the network itself.

  1. Apply temperature scaling by dividing the logits by the temperature value.
  2. Select the k largest logits and set every other logit to negative infinity.
  3. Apply softmax so that the k surviving tokens form a renormalized distribution.
  4. Draw one token at random according to those probabilities and append it to the sequence.

The following table shows the effect on a simplified distribution after the prompt “The capital of France is”, with k = 3.

Candidate tokenOriginal probabilityKept with k = 3Renormalized probability
Paris0.82Yes0.87
the0.07Yes0.07
a0.05Yes0.05
Lyon0.03No0.00
All remaining tokens0.03No0.00
Example of truncation and renormalization with k = 3

Choosing the value of k

The value of k sets a fixed upper bound on diversity: small values make output conservative and predictable, while large values let rarer words appear. When k = 1, top-k sampling is mathematically identical to greedy decoding, and when k equals the vocabulary size, the filter has no effect at all.

Practical defaults differ between tasks and between serving stacks, so the ranges below are typical starting points rather than fixed rules. Tuning should be done together with temperature, because both parameters shape the same distribution.

Value of kBehaviorTypical use
1Identical to greedy decodingDeterministic extraction, classification
5-20Focused, low varietyFactual answers, code completion
40-50Balanced variety and coherenceGeneral chat, drafting
100 or moreHigh variety, more off-topic riskBrainstorming, fiction
0 or disabledNo truncation appliedWhen top-p or min-p does the filtering
Typical k ranges and their effect on generated text

Note that in several popular libraries a value of k = 0 means “disabled”, not “no tokens”, which is a frequent source of confusion when settings are copied between tools.

Top-k compared with other decoding strategies

Top-k sampling differs from its neighbors mainly in how the candidate set is chosen. Top-k uses a fixed count, top-p (nucleus) sampling uses a cumulative probability mass, and min-p sampling uses a threshold relative to the most likely token.

StrategyCandidate setAdapts to model confidenceMain weakness
Greedy decodingSingle most probable tokenNoRepetitive, bland text
Beam searchSeveral best partial sequencesNoGeneric output in open-ended tasks
Top-k samplingFixed number of best tokensNoSame k for flat and peaked distributions
Top-p samplingSmallest set reaching mass pYesCan include many weak tokens when flat
Min-p samplingTokens above a fraction of the top probabilityYesNewer, less standardized support
How common decoding strategies select the next token

The central weakness of top-k is that it is blind to the shape of the distribution. When the model is very confident, as with “Paris” above, k = 40 still admits 39 unlikely alternatives; when the model is uncertain among hundreds of reasonable words, k = 40 may cut off valid continuations. Nucleus sampling was proposed largely to address this problem.

Combining top-k with temperature and other filters

In production systems top-k is rarely used alone; it is one stage in a chain of logit processors. The order of these stages matters, and it is not identical across frameworks, so the same numeric settings can produce different behavior in different serving engines.

  • Repetition and frequency penalties usually adjust logits before any truncation.
  • Temperature reshapes the distribution; applied before top-k, it does not change which tokens survive, only their relative weights.
  • Top-k followed by top-p acts as a safety cap: top-p adapts to confidence, while top-k guarantees that no more than k tokens are ever considered.
  • Constrained decoding masks tokens that violate a grammar or schema, and top-k then operates only on the valid remainder.

Some hosted model APIs expose top-k, others expose only temperature and top-p, and a few reasoning models fix sampling parameters internally. Checking the provider documentation before tuning avoids changing a parameter that is silently ignored.

Limitations and common mistakes

Top-k sampling is simple and fast, but its fixed cutoff creates predictable failure modes. Most problems come from treating k as a quality dial rather than as a blunt safety limit.

SymptomLikely causeFix
Repetitive, flat answersk too small or temperature too lowRaise k moderately or raise temperature
Sudden off-topic wordsk too large with high temperatureLower k or add top-p around 0.9
Different output after migrationk = 0 meaning differs by toolVerify the default and disabled value
Invalid JSON despite low kSampling cannot enforce structureUse constrained decoding
Run-to-run variation in testsRandom draw without fixed seedFix the seed or use greedy decoding
Frequent problems with top-k settings and how to fix them

A common misconception is that a low k prevents hallucination. Truncation only removes unlikely tokens; when the model assigns high probability to a wrong fact, top-k will select it just as readily as a correct one. Factual reliability depends on the model and the context, not on the sampler.

Understanding it simply

A language model writes one small piece of text at a time. For each piece, it ranks thousands of possible next words by how likely they are. Top-k sampling tells the model to look only at its best k options and then pick one of them at random, with more likely words getting a bigger chance.

Think of a quiz contestant who writes down the five answers that feel most plausible and then draws one from a hat weighted toward the strongest guess. The answer still varies, but nonsense is never on the shortlist. A small k gives safe, predictable text; a larger k gives more variety and a little more risk.

Applied in practice

A team at an online retailer uses an open-weight model to write short product descriptions for 8,000 catalog items. With pure sampling at temperature 1.0, roughly one description in twenty contains an odd word or a sentence that drifts away from the product. With greedy decoding, the descriptions become nearly identical and start every item with the same phrase.

The engineers settle on temperature 0.8, top-k of 40 and top-p of 0.92. Top-p adapts to how certain the model is, and top-k caps the shortlist so that no rare token slips in when the distribution is flat. In a review of 300 samples, editors find that off-topic phrasing almost disappears while wording still varies enough that the pages do not read as duplicates. For the structured specification table on each page, the team switches to greedy decoding with a JSON schema, because variety is unwanted where exact fields matter.