Greedy decoding is the text generation method in which a language model always selects the single most probable next token at each step, appends it to the sequence and repeats until a stop condition is met. In language models used across artificial intelligence, it is the simplest and fully deterministic decoding strategy: the same prompt and the same model weights produce the same output, apart from minor numerical effects of hardware and batching.
Greedy decoding is equivalent to sampling with a temperature of zero, and it serves as the reference point against which beam search, top-k sampling, top-p sampling and min-p sampling are described. It is fast and predictable, but it optimizes each step locally rather than the whole sequence, which explains both its strengths and its well-known failure modes.
How greedy decoding selects each token
Greedy decoding selects each token by taking the argmax of the logits that the model produces for the next position. An autoregressive model reads the prompt plus everything generated so far, computes one logit per vocabulary entry, and the decoder picks the index with the highest value. Applying softmax first is unnecessary, because softmax preserves the ranking of the logits.
The chosen token is appended to the context, and the loop repeats. Generation ends when the model emits an end-of-sequence token, when a configured stop sequence appears, or when the max tokens limit is reached. Because only one candidate survives each step, the method needs no extra memory for alternative hypotheses and adds negligible computational overhead beyond the forward pass itself.
The table below traces three steps of a hypothetical generation and shows how a single winning token is kept at each position while all alternatives are discarded.
| Step | Context so far | Top candidates and probabilities | Token kept |
|---|---|---|---|
| 1 | The capital of France | is 0.82, has 0.06, was 0.04 | is |
| 2 | The capital of France is | Paris 0.91, a 0.03, the 0.02 | Paris |
| 3 | The capital of France is Paris | . 0.64, , 0.21, and 0.08 | . |
Formally, greedy decoding approximates the most likely sequence by maximizing the conditional probability at each position. It does not guarantee the globally most probable sequence, because a token with slightly lower probability now may enable a much more probable continuation later.
Key properties and configuration
Greedy decoding has few parameters of its own, but several surrounding settings change its behavior in practice. Most inference APIs expose it implicitly through temperature set to 0 or an explicit flag that disables sampling.
| Setting | Typical value | Effect on greedy output |
|---|---|---|
| Temperature | 0 | Selects the argmax token; disables randomness |
| Repetition penalty | 1.0 to 1.3 | Lowers logits of tokens already used |
| Frequency penalty | 0 to 1.0 | Penalizes tokens in proportion to their count |
| Stop sequence | Task specific | Ends output at a defined string |
| Max tokens | Task specific | Caps length and prevents endless loops |
| Random seed | Irrelevant | Has no effect because no sampling occurs |
Determinism is not absolute. Floating-point arithmetic on GPUs is not associative, so parallel reductions, batch composition and kernel choice can change logits in the last decimal places. When two candidates are nearly tied, such a difference can flip the argmax and change everything that follows. Production systems that require bit-exact reproducibility pin hardware, batch size and software versions.
Greedy decoding compared with other decoding strategies
Greedy decoding differs from other strategies in how many candidates it keeps and whether it introduces randomness. Beam search keeps several partial sequences, while sampling methods draw tokens from a truncated or reshaped probability distribution.
| Strategy | Candidates kept | Deterministic | Typical use |
|---|---|---|---|
| Greedy decoding | One | Yes | Extraction, classification, code, evaluation |
| Beam search | Beam width, often 4 to 10 | Yes | Machine translation, summarization |
| Top-k sampling | k most probable tokens | No | Open-ended text with bounded variety |
| Top-p sampling | Smallest set reaching mass p | No | Chat and creative writing |
| Min-p sampling | Tokens above a relative threshold | No | Creative text at higher temperature |
| Constrained decoding | Only grammar-valid tokens | Optional | Structured output such as JSON |
Greedy decoding is a special case of beam search with a beam width of one and the limit of sampling as temperature approaches zero. It also combines with constrained decoding: masking invalid tokens and then taking the argmax is a common way to produce valid JSON or match a fixed label set.
When to use greedy decoding
Greedy decoding is the right choice when one correct answer exists and variety has no value. Modern instruction-tuned and reasoning models are usually trained so that their most probable continuation is also their best answer for factual and structured tasks.
| Situation | Recommended choice | Reason |
|---|---|---|
| Data extraction into fixed fields | Greedy decoding | Stable, repeatable output |
| Benchmark evaluation | Greedy decoding | Results comparable across runs |
| Code completion | Greedy or low temperature | Most probable token is usually correct |
| Brainstorming and fiction | Top-p or min-p sampling | Greedy output becomes bland and repetitive |
| Self-consistency voting | Sampling with several runs | Needs diverse reasoning paths |
| Long open-ended generation | Sampling with penalties | Greedy loops are more likely |
In evaluation pipelines, greedy decoding is often reported as pass@1 with temperature 0 because it removes sampling noise from comparisons between models. For reasoning models that spend test-time compute on long chains of thought, vendors sometimes recommend a moderate temperature instead, since purely greedy reasoning can get stuck in repetitive loops.
Limitations and common failure modes
The central weakness of greedy decoding is myopia: it commits to the locally best token and can never revisit that choice. Several characteristic problems follow from this property.
- Repetition loops: once a phrase becomes highly probable in its own context, the model may repeat it indefinitely, especially in base models and long outputs.
- Bland, generic text: the most probable phrasing is often the most common one, so creative output sounds flat and predictable.
- Early commitment errors: a slightly wrong first token, such as an opening word that implies a wrong answer, constrains everything after it.
- No diversity: repeated calls return identical answers, which makes greedy decoding unsuitable for generating alternatives or for majority voting.
- No protection against hallucination: a confident argmax token can still be false, so greedy output is not automatically more factual.
A frequent misconception holds that greedy decoding returns the model’s single best answer. It returns the sequence built from the best token at each step, which may differ from both the most probable full sequence and the most useful one. Repetition penalties, stop sequences and a sensible max tokens value mitigate the most practical failure modes without giving up determinism.
A plain-language summary
A language model writes text one small piece at a time. At each point it ranks every possible next word piece by how likely it seems. Greedy decoding simply takes the top-ranked piece every time and never looks back.
Think of a hiker who always takes the steepest path uphill at each fork. The hiker moves quickly and predictably, but may end up on a small hill instead of the highest peak. In the same way, greedy decoding is fast and repeatable, which suits tasks with one right answer, yet it can produce dull or repetitive text when creativity matters.
An example from practice
An insurance company processes about 20,000 scanned claim letters per month with a large language model that extracts the policy number, claim date, damage category and requested amount into a JSON record. The engineering team runs the model with temperature 0, a JSON grammar enforced by constrained decoding, a stop sequence after the closing brace and a limit of 300 output tokens.
Because greedy decoding is deterministic, auditors can rerun any disputed letter and receive the same extraction, and the team can compare model versions on a fixed test set of 1,500 letters without sampling noise. An earlier trial with top-p sampling at 0.9 produced different damage categories for about 3 percent of letters on repeated runs. With greedy decoding, the output became fully reproducible, and the remaining errors could be traced to genuinely ambiguous letters rather than to random variation.