Prefill phase

Also known as: prefill stage

On this page 7

The prefill phase, also called the prefill stage, is the first stage of large language model inference, in which the model processes all prompt tokens in parallel in a single forward pass, fills the KV cache with keys and values for every layer, and computes the probability distribution for the first output token. In AI infrastructure, it is one of the two fundamental phases of autoregressive serving, followed by the decode phase, which generates the remaining tokens one at a time.

The two phases stress hardware in opposite ways. Prefill is compute-bound, dominated by large matrix multiplications, while decode is memory-bandwidth-bound. This asymmetry shapes scheduling, batching, hardware choice and the user-visible metric time to first token.

What happens during the prefill phase

During prefill, the inference server tokenizes the prompt, embeds every token and runs the full stack of transformer layers over the entire sequence at once. Because all prompt tokens are known in advance, the attention and feed-forward computations become dense matrix-matrix operations that keep GPU tensor cores busy.

Each attention layer computes queries, keys and values for every prompt position. The keys and values are stored in the KV cache, so the decode phase never has to recompute them. Only the hidden state of the last prompt position is needed to sample the first generated token.

The following table summarizes the main operations inside a prefill pass and their cost behavior.

OperationPurposeCost scaling with prompt length
Tokenization and embeddingTurn text into vectorsLinear, negligible on GPU
QKV projectionsCreate queries, keys and valuesLinear
Self-attentionRelate every token to earlier tokensQuadratic in sequence length
Feed-forward layersTransform each positionLinear, largest share for short prompts
KV cache writeStore keys and values for decodeLinear in memory
Final logitsScore the first output tokenConstant, last position only
Operations performed during prefill

Prefill phase versus decode phase

The prefill phase and the decode phase run the same model weights but have fundamentally different performance profiles. Understanding the contrast is the basis of nearly every modern optimization in model serving.

PropertyPrefill phaseDecode phase
Tokens per forward passEntire promptOne per sequence
ParallelismAcross all prompt positionsOnly across batched requests
Hardware bottleneckCompute (FLOPs)Memory bandwidth
Arithmetic intensityHighLow
Latency metricTime to first tokenTime per output token
KV cache roleWrittenRead and appended
Comparison of the two inference phases

For a prompt of a few thousand tokens, prefill can reach high model FLOPs utilization, often well above half of peak on modern accelerators, whereas decode at small batch sizes typically uses only a small fraction of peak compute. A long prompt therefore costs fewer seconds per token than generated output, which is why many providers price input tokens below output tokens.

Factors that determine prefill latency

Prefill latency, experienced by the user as time to first token (TTFT), grows with the amount of computation required and with any queueing before the request is scheduled. The dominant factors are listed below.

FactorEffect on prefillTypical mitigation
Prompt lengthRoughly linear, quadratic attention at long contextsPrompt caching, shorter context
Model sizeProportional to parameter countSmaller or distilled model
Attention implementationMemory traffic of attention matrixFlashAttention kernels
Batch compositionLong prefills delay other requestsChunked prefill
Parallelism strategySplits compute across GPUsTensor or context parallelism
Numeric precisionLower precision raises throughputFP8 or INT8 inference
Main drivers of prefill cost

For short prompts, feed-forward layers dominate cost; as prompts grow into tens of thousands of tokens, the quadratic attention term becomes significant. At very long contexts, context parallelism or sequence parallelism splits a single prefill across several devices to keep TTFT acceptable.

Optimization techniques for the prefill stage

Serving systems apply several techniques specifically to make prefill faster or to stop it from interfering with decode traffic. Most are available in open-source engines such as vLLM and in commercial inference servers.

  • Prompt caching (prefix caching) reuses the KV cache of a shared prefix, such as a system prompt, so only new tokens are prefilled.
  • Chunked prefill splits a long prompt into fixed-size chunks and interleaves them with decode steps, smoothing latency for other users.
  • Disaggregated serving runs prefill and decode on separate GPU pools and transfers the KV cache between them.
  • FlashAttention computes attention in tiles held in on-chip memory, avoiding materialization of the full attention matrix.
  • Continuous batching admits new prefills into running batches without waiting for earlier sequences to finish.

The choice involves a clear trade-off. Prioritizing prefill lowers TTFT for new requests but causes visible stalls in ongoing streams, while prioritizing decode keeps streaming smooth at the cost of longer waits before the first token. Chunked prefill and disaggregation exist to balance these two objectives.

Common misconceptions and operational pitfalls

Several misunderstandings about prefill lead to poor capacity planning and disappointing latency in production. The most frequent ones are summarized here.

  • Prefill is not free: input tokens are cheaper per token, yet a 100,000-token prompt can take seconds before any output appears.
  • TTFT is not pure prefill time: queueing, tokenization and network delay are included in the measured value.
  • Long-context benchmarks mislead: averages hide tail latency caused by occasional very long prompts.
  • KV cache memory is set at prefill: long prompts reserve memory that limits how many sequences can decode concurrently.
  • Cache hits depend on exact prefixes: changing one early token, such as a timestamp in the system prompt, invalidates prompt caching.

Effective monitoring separates prefill throughput in tokens per second from decode throughput and tracks TTFT percentiles, not only means, through LLM observability tooling. Placing stable content first in prompts maximizes prefix cache reuse.

A plain-language summary

Before a language model can answer, it has to read the whole question. The prefill phase is that reading step. The model looks at every word of the prompt at the same time and takes notes it will use while writing the answer.

Reading is fast because it happens all at once, while writing the answer happens word by word. Long prompts mean a longer wait before the first word of the reply appears, even when the answer itself is short.

An example from practice

A company runs an internal assistant that answers questions about policy documents. Each request carries a 6,000-token system prompt with instructions and reference text, followed by a short user question of about 50 tokens. Users complain that the first word of each answer takes almost two seconds to appear during busy hours.

The infrastructure team finds that prefill dominates time to first token, and that long prompts block decode steps of other users. They move all stable content to the start of the prompt and enable prefix caching, so only the user question is prefilled for most requests. They also turn on chunked prefill with 512-token chunks.

Median time to first token falls to a few hundred milliseconds, GPU compute per request drops sharply, and streaming stays smooth for concurrent users because large prefills no longer monopolize the accelerator.