A decoding strategy is the algorithm that converts a model’s next-token probability distribution into a concrete sequence of tokens. In language models, the network itself only scores every possible continuation; the decoding strategy, also called a sampling strategy, decides which token is actually emitted at each step and therefore shapes fluency, diversity, accuracy and cost of the generated text.
Because autoregressive models generate one token at a time and feed each choice back as input, small decoding choices compound over hundreds of steps. The same trained weights can produce terse factual answers, repetitive loops or creative prose depending only on the strategy and its parameters.
How a decoding strategy turns logits into text
A decoding strategy operates on the logits, the raw scores the model outputs for every token in its vocabulary at each position. These scores are usually divided by a temperature value and passed through a softmax function, which yields a probability distribution summing to one.
The strategy then applies three kinds of operations. It may reshape the distribution (temperature, repetition or frequency penalties), truncate the candidate set (top-k, top-p, min-p), and finally select a token, either deterministically by taking the maximum or stochastically by sampling. Each selected token becomes part of the input for the following step, and generation halts once the model emits an end-of-sequence token, matches a stop sequence, or exhausts the max tokens budget.
The following table lists the stages that a typical inference pipeline applies, in the order most implementations use.
| Stage | Operation | Effect on output |
|---|---|---|
| Penalties | Lower logits of tokens already used | Reduces loops and repetition |
| Temperature | Divide logits by a constant | Sharpens or flattens the distribution |
| Truncation | Remove unlikely candidates | Cuts off the incoherent tail |
| Constraints | Mask tokens that break a grammar | Guarantees valid structure |
| Selection | Argmax or random draw | Emits the next token |
Deterministic and stochastic decoding families
Decoding strategies fall into two broad families: deterministic search, which always returns the same output for the same input, and stochastic sampling, which draws tokens at random from a filtered distribution. Search methods approximate the most probable sequence; sampling methods trade some probability mass for diversity and naturalness.
The table compares the most widely used strategies and the parameters that control them.
| Strategy | Mechanism | Key parameter | Typical use |
|---|---|---|---|
| Greedy decoding | Always picks the highest-probability token | None | Short factual answers, classification |
| Beam search | Keeps several partial sequences and expands the best | Beam width, often 4 to 10 | Machine translation, summarization |
| Pure sampling | Draws from the full distribution | Temperature | Rarely used alone |
| Top-k sampling | Samples from the k most likely tokens | k, often 20 to 100 | Open-ended generation |
| Top-p sampling | Samples from the smallest set reaching probability p | p, often 0.9 to 0.95 | Chat assistants, creative writing |
| Min-p sampling | Keeps tokens above a fraction of the top probability | Min-p, often 0.05 to 0.1 | High-temperature creative output |
| Constrained decoding | Masks tokens that violate a schema or grammar | Grammar or JSON schema | Structured output, tool calls |
Beam search maximizes sequence-level likelihood rather than per-token likelihood, which helps tasks with a narrow set of correct outputs. In open-ended generation, however, the most probable sequence is often bland and repetitive, a well-documented effect sometimes called the likelihood trap.
Parameters that shape the output distribution
Most production APIs expose a decoding strategy as a set of sampling parameters rather than a single named method. Setting temperature to zero, for example, effectively reduces sampling to greedy decoding, while combining temperature with top-p yields nucleus sampling with adjustable randomness.
| Parameter | Common range | Low value effect | High value effect |
|---|---|---|---|
| Temperature | 0 to 2 | Focused, predictable text | Varied, riskier text |
| Top-p | 0.1 to 1.0 | Few candidates, conservative | Most candidates, diverse |
| Top-k | 1 to vocabulary size | Near-greedy behavior | Little truncation |
| Repetition penalty | 1.0 to 1.3 | Possible loops | Avoids reused words, may harm grammar |
| Frequency penalty | 0 to 2 | No adjustment | Strongly discourages frequent tokens |
Parameters interact rather than add up. A high temperature with a strict top-p can behave more conservatively than a moderate temperature with no truncation, because truncation removes the tail before the random draw. Providers also differ in the order in which they apply these operations, so identical numbers can produce different behavior across inference engines.
Choosing a decoding strategy for a task
The right decoding strategy depends on whether the task has one correct answer or many acceptable ones. Tasks with verifiable outputs benefit from low randomness, while tasks that reward variety benefit from controlled sampling.
| Task | Recommended setting | Reason |
|---|---|---|
| Data extraction | Greedy plus constrained decoding | Output must match a schema exactly |
| Code generation | Temperature 0 to 0.3 | Small deviations break syntax |
| Customer support chat | Temperature 0.3 to 0.7, top-p 0.9 | Natural tone with stable facts |
| Brainstorming and fiction | Temperature 0.8 to 1.2, min-p or top-p | Rewards novelty and variety |
| Best-of-n reasoning | Moderate temperature, several samples | Diverse candidates for voting or verification |
With reasoning models and test-time compute techniques, decoding becomes part of the reasoning method itself. Self-consistency, for example, samples several chain-of-thought paths at a nonzero temperature and selects the most frequent final answer, which only works when sampling produces genuinely different paths.
Limitations and common misconceptions
A decoding strategy cannot add knowledge the model does not have. It redistributes probability among continuations the model already considers plausible, so it changes style and risk but not the underlying parametric knowledge.
- Temperature zero is not fully deterministic in practice; batching, floating-point nondeterminism and parallel hardware can still change outputs slightly.
- Lower randomness does not eliminate hallucination; a model can confidently produce a wrong answer under greedy decoding.
- Beam search is not universally better; for chat and long-form writing it often amplifies repetition.
- Penalties can damage correctness when legitimate repetition is required, such as variable names in code or terms in legal text.
- Settings do not transfer between models; a well-calibrated value for one model may be too hot or too cold for another.
Decoding cost also matters. Beam search multiplies computation by the beam width, and best-of-n sampling multiplies it by n, so teams should measure whether the quality gain justifies latency and token spend.
A plain-language summary
A language model does not write a sentence in one go. At every step it produces a ranked list of possible next words, each with a likelihood. The decoding strategy is the rule for picking one word from that list.
One rule always takes the top choice, which is safe but can sound flat. Another rule rolls weighted dice among the best options, which sounds more natural and creative but sometimes wanders. Choosing the rule is like choosing a cautious or adventurous driver for the same car: the engine stays identical, but the route changes.
An example from practice
A software team builds an assistant that reads incoming invoices and fills an accounting form with supplier name, date, amount and tax rate. In early tests with temperature 0.8 and top-p 0.95, about one output in twenty contains a malformed date or an invented field, which breaks the import job.
The team switches to greedy decoding combined with constrained decoding against a JSON schema, so the model can only emit valid keys and correctly formatted values. Malformed records disappear, and remaining errors are limited to genuinely unreadable scans. For the marketing team’s product descriptions, the same model keeps temperature 0.9 with min-p 0.05, producing varied copy. One model serves both teams; only the decoding strategy differs.