Speculative decoding is an inference technique in which a small, fast draft model proposes several tokens ahead and a large target model verifies them in a single forward pass, accepting the correct prefix and correcting the first wrong token. In AI infrastructure, it is one of the most effective ways to reduce time per output token in large language model serving without changing the output distribution.
The method exploits a simple asymmetry: generating tokens one at a time is limited by memory bandwidth rather than compute, so checking several candidate tokens at once costs almost the same as producing one. When the draft guesses well, each expensive target pass yields multiple tokens instead of one.
How speculative decoding works step by step
Speculative decoding works as a repeated draft, verify and accept loop. Each iteration produces at least one token and often several, while the target model remains the final authority on every token that reaches the user.
- The draft model generates a short sequence of k candidate tokens autoregressively, typically 3 to 8 tokens.
- The target model processes the original context plus all k candidates in one forward pass, producing its own probability distribution at every position.
- Each candidate is compared with the target distribution and accepted or rejected by a rule based on the ratio of target to draft probabilities.
- At the first rejection, the target samples a replacement token from an adjusted distribution and the remaining candidates are discarded.
- If all k candidates are accepted, the target pass also supplies one bonus token, and the loop starts again.
The acceptance rule, known as speculative sampling, is mathematically constructed so that the final tokens follow exactly the target model distribution. With greedy decoding the rule reduces to a simple check that each draft token equals the target top choice. The technique is therefore lossless in quality, unlike quantization or distillation, which change the model itself.
Key parameters and their effect on speedup
The speedup from speculative decoding depends mainly on the acceptance rate, the draft length and the relative cost of the draft model. The table below summarizes these parameters and typical ranges reported in practice.
| Parameter | Typical range | Effect |
|---|---|---|
| Acceptance rate | 0.5 to 0.9 per token | Higher rate means more tokens per target pass |
| Draft length k | 3 to 8 tokens | Longer drafts help only when acceptance is high |
| Draft cost ratio | 1 to 10 percent of target cost | Expensive drafts erase the gain |
| Sampling temperature | 0 to 1 | Higher temperature usually lowers acceptance |
| Batch size | 1 to hundreds of sequences | Gains shrink as the server becomes compute-bound |
Under favorable conditions, such as code completion or structured output at low temperature, end-to-end latency typically improves by a factor of 2 to 3. For creative text at high temperature the gain may fall to 1.2 to 1.5 times, and a poorly matched draft can make generation slower than plain decoding.
Variants of the draft mechanism
Variants of speculative decoding differ mainly in where the draft tokens come from. A separate small model is the original design, but several approaches avoid the cost of hosting a second network.
| Variant | Draft source | Advantage | Drawback |
|---|---|---|---|
| Separate draft model | Smaller model of the same family | Simple, no target retraining | Extra memory, shared tokenizer required |
| Extra prediction heads | Additional heads on the target, as in Medusa | No second model to serve | Heads must be trained |
| Feature-level drafting | Light layer reusing target hidden states, as in EAGLE | High acceptance rates | Model-specific training |
| Prompt lookup decoding | N-gram matches copied from the prompt | Zero training, near-zero cost | Helps only when output repeats input |
| Self-speculative decoding | Target with skipped layers | Single set of weights | Lower acceptance than trained drafts |
Tree-based verification extends all variants: the draft proposes several branching continuations, and the target verifies the whole tree with a special attention mask, raising the expected number of accepted tokens per pass.
Speculative decoding compared with other inference optimizations
Speculative decoding targets per-request latency in the decode phase, while many other optimizations target throughput or memory. Understanding the differences helps teams combine them correctly.
| Technique | Main target | Changes output | Interaction with speculative decoding |
|---|---|---|---|
| Speculative decoding | Decode latency | No | Reference technique |
| Quantization | Memory and bandwidth | Slightly | Complementary, often applied to both models |
| Continuous batching | Throughput | No | Large batches reduce the speculative gain |
| KV cache optimizations | Memory per sequence | No | Rejected tokens require cache rollback |
| Distillation | Model size | Yes | Distilled models make good drafts |
The interaction with batching is the most important trade-off. Large batches already saturate GPU compute, so verifying extra speculative tokens consumes capacity that would otherwise serve other users. Many serving stacks therefore enable speculation dynamically at low load and reduce or disable it at high load.
Limitations and practical pitfalls
The main limitation of speculative decoding is that its benefit is workload-dependent and hard to predict without measurement. Teams that enable it by default sometimes see higher cost per token with no latency gain.
- Tokenizer mismatch: a draft model must share the target vocabulary, or tokens must be mapped, which reduces acceptance.
- Memory overhead: a separate draft model and its own KV cache consume GPU memory that could hold more sequences.
- Wasted compute: rejected tokens are computed and discarded, which raises energy use per accepted token.
- Distribution drift: a draft tuned on different data than the live traffic shows falling acceptance over time.
- Measurement errors: benchmarking only at batch size 1 overstates the gain seen in a busy production server.
A sound practice is to monitor the acceptance rate as a production metric alongside time to first token and tokens per second, and to re-evaluate the draft whenever the target model or traffic pattern changes.
Put simply
Speculative decoding is like a junior writer drafting the next few words of a sentence while a senior editor checks them all at once. The editor keeps every word that is right, fixes the first wrong one and asks for a new draft from there.
Because checking a short draft takes the editor almost the same time as writing a single word, the text appears much faster. The final result is exactly what the editor would have written alone, only quicker. When the junior writer guesses poorly, the trick stops paying off.
In practice
A software company runs a 70-billion-parameter model to power code completion inside its developers’ editors. Users expect suggestions within a few hundred milliseconds, but plain decoding produces only about 25 tokens per second per request, which makes multi-line suggestions feel sluggish.
The infrastructure team adds a 1-billion-parameter draft model from the same family, drafting five tokens per step at temperature 0. Because code contains predictable syntax and repeated identifiers, the measured acceptance rate reaches about 0.8. Output speed rises to roughly 60 tokens per second per request, and median suggestion latency drops by more than half. During peak hours, when batches grow large, the serving layer automatically shortens drafts to two tokens, keeping overall GPU throughput stable while preserving most of the latency gain.