Speculative decoding

On this page 7

Speculative decoding is an inference technique in which a small, fast draft model proposes several tokens ahead and a large target model verifies them in a single forward pass, accepting the correct prefix and correcting the first wrong token. In AI infrastructure, it is one of the most effective ways to reduce time per output token in large language model serving without changing the output distribution.

The method exploits a simple asymmetry: generating tokens one at a time is limited by memory bandwidth rather than compute, so checking several candidate tokens at once costs almost the same as producing one. When the draft guesses well, each expensive target pass yields multiple tokens instead of one.

How speculative decoding works step by step

Speculative decoding works as a repeated draft, verify and accept loop. Each iteration produces at least one token and often several, while the target model remains the final authority on every token that reaches the user.

  1. The draft model generates a short sequence of k candidate tokens autoregressively, typically 3 to 8 tokens.
  2. The target model processes the original context plus all k candidates in one forward pass, producing its own probability distribution at every position.
  3. Each candidate is compared with the target distribution and accepted or rejected by a rule based on the ratio of target to draft probabilities.
  4. At the first rejection, the target samples a replacement token from an adjusted distribution and the remaining candidates are discarded.
  5. If all k candidates are accepted, the target pass also supplies one bonus token, and the loop starts again.

The acceptance rule, known as speculative sampling, is mathematically constructed so that the final tokens follow exactly the target model distribution. With greedy decoding the rule reduces to a simple check that each draft token equals the target top choice. The technique is therefore lossless in quality, unlike quantization or distillation, which change the model itself.

Key parameters and their effect on speedup

The speedup from speculative decoding depends mainly on the acceptance rate, the draft length and the relative cost of the draft model. The table below summarizes these parameters and typical ranges reported in practice.

ParameterTypical rangeEffect
Acceptance rate0.5 to 0.9 per tokenHigher rate means more tokens per target pass
Draft length k3 to 8 tokensLonger drafts help only when acceptance is high
Draft cost ratio1 to 10 percent of target costExpensive drafts erase the gain
Sampling temperature0 to 1Higher temperature usually lowers acceptance
Batch size1 to hundreds of sequencesGains shrink as the server becomes compute-bound
Parameters that govern speculative decoding performance

Under favorable conditions, such as code completion or structured output at low temperature, end-to-end latency typically improves by a factor of 2 to 3. For creative text at high temperature the gain may fall to 1.2 to 1.5 times, and a poorly matched draft can make generation slower than plain decoding.

Variants of the draft mechanism

Variants of speculative decoding differ mainly in where the draft tokens come from. A separate small model is the original design, but several approaches avoid the cost of hosting a second network.

VariantDraft sourceAdvantageDrawback
Separate draft modelSmaller model of the same familySimple, no target retrainingExtra memory, shared tokenizer required
Extra prediction headsAdditional heads on the target, as in MedusaNo second model to serveHeads must be trained
Feature-level draftingLight layer reusing target hidden states, as in EAGLEHigh acceptance ratesModel-specific training
Prompt lookup decodingN-gram matches copied from the promptZero training, near-zero costHelps only when output repeats input
Self-speculative decodingTarget with skipped layersSingle set of weightsLower acceptance than trained drafts
Common speculative decoding variants

Tree-based verification extends all variants: the draft proposes several branching continuations, and the target verifies the whole tree with a special attention mask, raising the expected number of accepted tokens per pass.

Speculative decoding compared with other inference optimizations

Speculative decoding targets per-request latency in the decode phase, while many other optimizations target throughput or memory. Understanding the differences helps teams combine them correctly.

TechniqueMain targetChanges outputInteraction with speculative decoding
Speculative decodingDecode latencyNoReference technique
QuantizationMemory and bandwidthSlightlyComplementary, often applied to both models
Continuous batchingThroughputNoLarge batches reduce the speculative gain
KV cache optimizationsMemory per sequenceNoRejected tokens require cache rollback
DistillationModel sizeYesDistilled models make good drafts
Comparison with neighboring optimization techniques

The interaction with batching is the most important trade-off. Large batches already saturate GPU compute, so verifying extra speculative tokens consumes capacity that would otherwise serve other users. Many serving stacks therefore enable speculation dynamically at low load and reduce or disable it at high load.

Limitations and practical pitfalls

The main limitation of speculative decoding is that its benefit is workload-dependent and hard to predict without measurement. Teams that enable it by default sometimes see higher cost per token with no latency gain.

  • Tokenizer mismatch: a draft model must share the target vocabulary, or tokens must be mapped, which reduces acceptance.
  • Memory overhead: a separate draft model and its own KV cache consume GPU memory that could hold more sequences.
  • Wasted compute: rejected tokens are computed and discarded, which raises energy use per accepted token.
  • Distribution drift: a draft tuned on different data than the live traffic shows falling acceptance over time.
  • Measurement errors: benchmarking only at batch size 1 overstates the gain seen in a busy production server.

A sound practice is to monitor the acceptance rate as a production metric alongside time to first token and tokens per second, and to re-evaluate the draft whenever the target model or traffic pattern changes.

Put simply

Speculative decoding is like a junior writer drafting the next few words of a sentence while a senior editor checks them all at once. The editor keeps every word that is right, fixes the first wrong one and asks for a new draft from there.

Because checking a short draft takes the editor almost the same time as writing a single word, the text appears much faster. The final result is exactly what the editor would have written alone, only quicker. When the junior writer guesses poorly, the trick stops paying off.

In practice

A software company runs a 70-billion-parameter model to power code completion inside its developers’ editors. Users expect suggestions within a few hundred milliseconds, but plain decoding produces only about 25 tokens per second per request, which makes multi-line suggestions feel sluggish.

The infrastructure team adds a 1-billion-parameter draft model from the same family, drafting five tokens per step at temperature 0. Because code contains predictable syntax and repeated identifiers, the measured acceptance rate reaches about 0.8. Output speed rises to roughly 60 tokens per second per request, and median suggestion latency drops by more than half. During peak hours, when batches grow large, the serving layer automatically shortens drafts to two tokens, keeping overall GPU throughput stable while preserving most of the latency gain.