Disaggregated serving

Also known as: prefill-decode disaggregation

On this page 7

Disaggregated serving, also called prefill-decode disaggregation, is an inference architecture that runs the prefill phase and the decode phase of a large language model request on separate pools of accelerators. In AI infrastructure, it addresses a basic mismatch: prompt processing is compute-bound, while token-by-token generation is memory-bandwidth-bound, so one GPU configuration cannot serve both phases efficiently.

Instead of letting long prompts and ongoing generations compete on the same device, the system computes the KV cache on prefill workers, transfers it over a fast interconnect to decode workers and continues generation there. The result is more predictable latency and better use of hardware at scale, at the cost of extra network traffic and operational complexity.

Why prefill and decode behave so differently

The two phases of transformer inference stress hardware in opposite ways. Prefill processes all prompt tokens in parallel, performing large matrix multiplications that keep tensor cores busy. Decode produces one token per step for each sequence, reading the full model weights and the growing KV cache from memory for very little arithmetic.

PropertyPrefill phaseDecode phase
Main bottleneckCompute throughput (FLOPs)Memory bandwidth
Arithmetic intensityHigh, hundreds of FLOPs per byteLow, often near one per byte without batching
Latency metric affectedTime to first tokenTime per output token
Work per requestOne pass over the whole promptOne step per generated token
Benefit from large batchesLimited, already saturates computeLarge, amortizes weight reads
Characteristics of the two inference phases

When both phases share one GPU, a long prefill stalls every decode step queued behind it, producing visible pauses in streamed output. Techniques such as chunked prefill soften this interference, but they still force one batch size and one parallelism layout on two workloads with different needs.

How a disaggregated serving pipeline works

A disaggregated deployment splits the serving path into specialized stages connected by a KV cache transfer. A request travels through the stages in a fixed order.

  1. A router or scheduler receives the request and selects a prefill worker, often considering prefix cache hits.
  2. The prefill worker runs the full prompt through the model and produces the first token plus the KV cache for every layer.
  3. The KV cache is transferred to a chosen decode worker over NVLink, InfiniBand or RDMA-capable Ethernet.
  4. The decode worker adds the sequence to its continuous batch and generates the remaining tokens.
  5. Completed tokens stream back to the client, and the decode worker frees the cache blocks when generation ends.

Transfer cost is the central engineering constraint. For a 70-billion-parameter model with grouped-query attention, the KV cache occupies roughly hundreds of kilobytes per token, so a 4,000-token prompt can require about one gigabyte or more to move. The transfer must finish in a few tens of milliseconds, which is why high-bandwidth interconnects and layer-by-layer streaming that overlaps with computation are standard design elements.

Benefits and trade-offs of separating the phases

Disaggregation lets operators tune each pool independently: prefill workers can use higher tensor parallelism for fast prompt processing, while decode workers use larger batches and more memory for throughput. The ratio of prefill to decode instances can follow the actual traffic mix rather than a compromise.

AspectColocated servingDisaggregated serving
Phase interferencePrefill delays decode stepsEliminated between pools
Latency controlShared trade-off for TTFT and TPOTSeparate targets per phase
Hardware choiceOne GPU type for all workDifferent GPU types per pool possible
Network demandMinimal between workersHeavy KV cache transfer
Operational complexityLow, one replica typeHigh, routing and pool balancing
Best fitSmall or mixed deploymentsLarge fleets with strict latency SLOs
Colocated versus disaggregated serving

The main gain is often described as higher goodput, meaning the number of requests per second that meet both latency targets. Research systems such as DistServe and Splitwise reported substantial goodput improvements under strict service level objectives, although the actual gain depends heavily on prompt and output lengths and on interconnect speed.

Implementations and deployment choices

Disaggregated serving moved from research to production between 2023 and 2025. Inference engines such as vLLM and SGLang added prefill-decode separation, and orchestration frameworks such as NVIDIA Dynamo and llm-d coordinate routing and KV cache movement across clusters. Large model providers have described similar designs for their own fleets.

Choosing whether to disaggregate depends on workload shape and scale. The following decision table summarizes common situations.

SituationRecommended approachReason
Long prompts, short outputsDisaggregate with more prefill workersPrefill dominates and interferes with decode
Short prompts, long outputsColocated with continuous batchingLittle prefill pressure to isolate
Strict TTFT and TPOT targetsDisaggregateEach phase gets its own latency budget
Single node or few GPUsColocated with chunked prefillTransfer overhead outweighs the benefit
Heavy prefix reuse in RAG or agentsDisaggregate with cache-aware routingShared KV cache reduces repeated prefill
When disaggregated serving pays off

Some deployments go further and disaggregate attention from feed-forward layers or offload KV cache to CPU memory and storage tiers. These variants follow the same principle: place each part of inference on the resource that limits it.

Risks and common mistakes

The architecture introduces failure modes that colocated serving does not have. Most problems come from underestimating transfer cost or from static pool sizing that ignores changing traffic.

  • Slow interconnects: moving the KV cache over standard TCP networking can erase the latency gains entirely.
  • Fixed prefill-to-decode ratios: traffic shifts during the day, and an unbalanced ratio leaves one pool idle while the other queues.
  • Ignoring prefix caching: routing without cache awareness repeats prefill work that could have been reused.
  • Weak observability: latency must be traced across both pools and the transfer step to find bottlenecks.
  • Premature adoption: small deployments gain little and pay the full complexity cost.

Stripped to the essentials

Answering a question with a language model happens in two steps. First the model reads the whole question at once, which needs heavy raw calculation. Then it writes the answer one word at a time, which mostly needs fast access to memory.

Disaggregated serving gives each step its own team of machines, like a restaurant where one kitchen preps ingredients and another finishes the dishes. Each team is set up for its own job, the prepared work is handed over quickly, and customers wait less, even when the restaurant is busy.

Where it is applied

A company runs an internal coding assistant on a 70-billion-parameter model. Developers paste prompts of 6,000 to 20,000 tokens that include source files, but most answers are only a few hundred tokens long. On colocated servers, large pasted files caused streamed answers to freeze for over a second whenever a new long prompt arrived.

The infrastructure team splits its 32 GPUs into 12 prefill and 20 decode workers, connected through InfiniBand, and adds a router that prefers workers holding a matching prefix cache. Prefill workers use higher tensor parallelism; decode workers run large continuous batches.

After the change, median time to first token falls noticeably and the generation pauses disappear, while the same hardware serves more concurrent users within the latency target. The team monitors the pool ratio weekly and shifts two GPUs to prefill during release weeks, when prompts grow longer.