Decoding strategy

Also known as: sampling strategy

On this page 7

A decoding strategy is the algorithm that converts a model’s next-token probability distribution into a concrete sequence of tokens. In language models, the network itself only scores every possible continuation; the decoding strategy, also called a sampling strategy, decides which token is actually emitted at each step and therefore shapes fluency, diversity, accuracy and cost of the generated text.

Because autoregressive models generate one token at a time and feed each choice back as input, small decoding choices compound over hundreds of steps. The same trained weights can produce terse factual answers, repetitive loops or creative prose depending only on the strategy and its parameters.

How a decoding strategy turns logits into text

A decoding strategy operates on the logits, the raw scores the model outputs for every token in its vocabulary at each position. These scores are usually divided by a temperature value and passed through a softmax function, which yields a probability distribution summing to one.

The strategy then applies three kinds of operations. It may reshape the distribution (temperature, repetition or frequency penalties), truncate the candidate set (top-k, top-p, min-p), and finally select a token, either deterministically by taking the maximum or stochastically by sampling. Each selected token becomes part of the input for the following step, and generation halts once the model emits an end-of-sequence token, matches a stop sequence, or exhausts the max tokens budget.

The following table lists the stages that a typical inference pipeline applies, in the order most implementations use.

StageOperationEffect on output
PenaltiesLower logits of tokens already usedReduces loops and repetition
TemperatureDivide logits by a constantSharpens or flattens the distribution
TruncationRemove unlikely candidatesCuts off the incoherent tail
ConstraintsMask tokens that break a grammarGuarantees valid structure
SelectionArgmax or random drawEmits the next token
Typical order of operations in one decoding step

Deterministic and stochastic decoding families

Decoding strategies fall into two broad families: deterministic search, which always returns the same output for the same input, and stochastic sampling, which draws tokens at random from a filtered distribution. Search methods approximate the most probable sequence; sampling methods trade some probability mass for diversity and naturalness.

The table compares the most widely used strategies and the parameters that control them.

StrategyMechanismKey parameterTypical use
Greedy decodingAlways picks the highest-probability tokenNoneShort factual answers, classification
Beam searchKeeps several partial sequences and expands the bestBeam width, often 4 to 10Machine translation, summarization
Pure samplingDraws from the full distributionTemperatureRarely used alone
Top-k samplingSamples from the k most likely tokensk, often 20 to 100Open-ended generation
Top-p samplingSamples from the smallest set reaching probability pp, often 0.9 to 0.95Chat assistants, creative writing
Min-p samplingKeeps tokens above a fraction of the top probabilityMin-p, often 0.05 to 0.1High-temperature creative output
Constrained decodingMasks tokens that violate a schema or grammarGrammar or JSON schemaStructured output, tool calls
Common decoding strategies compared

Beam search maximizes sequence-level likelihood rather than per-token likelihood, which helps tasks with a narrow set of correct outputs. In open-ended generation, however, the most probable sequence is often bland and repetitive, a well-documented effect sometimes called the likelihood trap.

Parameters that shape the output distribution

Most production APIs expose a decoding strategy as a set of sampling parameters rather than a single named method. Setting temperature to zero, for example, effectively reduces sampling to greedy decoding, while combining temperature with top-p yields nucleus sampling with adjustable randomness.

ParameterCommon rangeLow value effectHigh value effect
Temperature0 to 2Focused, predictable textVaried, riskier text
Top-p0.1 to 1.0Few candidates, conservativeMost candidates, diverse
Top-k1 to vocabulary sizeNear-greedy behaviorLittle truncation
Repetition penalty1.0 to 1.3Possible loopsAvoids reused words, may harm grammar
Frequency penalty0 to 2No adjustmentStrongly discourages frequent tokens
Sampling parameters and their practical ranges

Parameters interact rather than add up. A high temperature with a strict top-p can behave more conservatively than a moderate temperature with no truncation, because truncation removes the tail before the random draw. Providers also differ in the order in which they apply these operations, so identical numbers can produce different behavior across inference engines.

Choosing a decoding strategy for a task

The right decoding strategy depends on whether the task has one correct answer or many acceptable ones. Tasks with verifiable outputs benefit from low randomness, while tasks that reward variety benefit from controlled sampling.

TaskRecommended settingReason
Data extractionGreedy plus constrained decodingOutput must match a schema exactly
Code generationTemperature 0 to 0.3Small deviations break syntax
Customer support chatTemperature 0.3 to 0.7, top-p 0.9Natural tone with stable facts
Brainstorming and fictionTemperature 0.8 to 1.2, min-p or top-pRewards novelty and variety
Best-of-n reasoningModerate temperature, several samplesDiverse candidates for voting or verification
Decision guide for common tasks

With reasoning models and test-time compute techniques, decoding becomes part of the reasoning method itself. Self-consistency, for example, samples several chain-of-thought paths at a nonzero temperature and selects the most frequent final answer, which only works when sampling produces genuinely different paths.

Limitations and common misconceptions

A decoding strategy cannot add knowledge the model does not have. It redistributes probability among continuations the model already considers plausible, so it changes style and risk but not the underlying parametric knowledge.

  • Temperature zero is not fully deterministic in practice; batching, floating-point nondeterminism and parallel hardware can still change outputs slightly.
  • Lower randomness does not eliminate hallucination; a model can confidently produce a wrong answer under greedy decoding.
  • Beam search is not universally better; for chat and long-form writing it often amplifies repetition.
  • Penalties can damage correctness when legitimate repetition is required, such as variable names in code or terms in legal text.
  • Settings do not transfer between models; a well-calibrated value for one model may be too hot or too cold for another.

Decoding cost also matters. Beam search multiplies computation by the beam width, and best-of-n sampling multiplies it by n, so teams should measure whether the quality gain justifies latency and token spend.

A plain-language summary

A language model does not write a sentence in one go. At every step it produces a ranked list of possible next words, each with a likelihood. The decoding strategy is the rule for picking one word from that list.

One rule always takes the top choice, which is safe but can sound flat. Another rule rolls weighted dice among the best options, which sounds more natural and creative but sometimes wanders. Choosing the rule is like choosing a cautious or adventurous driver for the same car: the engine stays identical, but the route changes.

An example from practice

A software team builds an assistant that reads incoming invoices and fills an accounting form with supplier name, date, amount and tax rate. In early tests with temperature 0.8 and top-p 0.95, about one output in twenty contains a malformed date or an invented field, which breaks the import job.

The team switches to greedy decoding combined with constrained decoding against a JSON schema, so the model can only emit valid keys and correctly formatted values. Malformed records disappear, and remaining errors are limited to genuinely unreadable scans. For the marketing team’s product descriptions, the same model keeps temperature 0.9 with min-p 0.05, producing varied copy. One model serves both teams; only the decoding strategy differs.