Disaggregated serving, also called prefill-decode disaggregation, is an inference architecture that runs the prefill phase and the decode phase of a large language model request on separate pools of accelerators. In AI infrastructure, it addresses a basic mismatch: prompt processing is compute-bound, while token-by-token generation is memory-bandwidth-bound, so one GPU configuration cannot serve both phases efficiently.
Instead of letting long prompts and ongoing generations compete on the same device, the system computes the KV cache on prefill workers, transfers it over a fast interconnect to decode workers and continues generation there. The result is more predictable latency and better use of hardware at scale, at the cost of extra network traffic and operational complexity.
Why prefill and decode behave so differently
The two phases of transformer inference stress hardware in opposite ways. Prefill processes all prompt tokens in parallel, performing large matrix multiplications that keep tensor cores busy. Decode produces one token per step for each sequence, reading the full model weights and the growing KV cache from memory for very little arithmetic.
| Property | Prefill phase | Decode phase |
|---|---|---|
| Main bottleneck | Compute throughput (FLOPs) | Memory bandwidth |
| Arithmetic intensity | High, hundreds of FLOPs per byte | Low, often near one per byte without batching |
| Latency metric affected | Time to first token | Time per output token |
| Work per request | One pass over the whole prompt | One step per generated token |
| Benefit from large batches | Limited, already saturates compute | Large, amortizes weight reads |
When both phases share one GPU, a long prefill stalls every decode step queued behind it, producing visible pauses in streamed output. Techniques such as chunked prefill soften this interference, but they still force one batch size and one parallelism layout on two workloads with different needs.
How a disaggregated serving pipeline works
A disaggregated deployment splits the serving path into specialized stages connected by a KV cache transfer. A request travels through the stages in a fixed order.
- A router or scheduler receives the request and selects a prefill worker, often considering prefix cache hits.
- The prefill worker runs the full prompt through the model and produces the first token plus the KV cache for every layer.
- The KV cache is transferred to a chosen decode worker over NVLink, InfiniBand or RDMA-capable Ethernet.
- The decode worker adds the sequence to its continuous batch and generates the remaining tokens.
- Completed tokens stream back to the client, and the decode worker frees the cache blocks when generation ends.
Transfer cost is the central engineering constraint. For a 70-billion-parameter model with grouped-query attention, the KV cache occupies roughly hundreds of kilobytes per token, so a 4,000-token prompt can require about one gigabyte or more to move. The transfer must finish in a few tens of milliseconds, which is why high-bandwidth interconnects and layer-by-layer streaming that overlaps with computation are standard design elements.
Benefits and trade-offs of separating the phases
Disaggregation lets operators tune each pool independently: prefill workers can use higher tensor parallelism for fast prompt processing, while decode workers use larger batches and more memory for throughput. The ratio of prefill to decode instances can follow the actual traffic mix rather than a compromise.
| Aspect | Colocated serving | Disaggregated serving |
|---|---|---|
| Phase interference | Prefill delays decode steps | Eliminated between pools |
| Latency control | Shared trade-off for TTFT and TPOT | Separate targets per phase |
| Hardware choice | One GPU type for all work | Different GPU types per pool possible |
| Network demand | Minimal between workers | Heavy KV cache transfer |
| Operational complexity | Low, one replica type | High, routing and pool balancing |
| Best fit | Small or mixed deployments | Large fleets with strict latency SLOs |
The main gain is often described as higher goodput, meaning the number of requests per second that meet both latency targets. Research systems such as DistServe and Splitwise reported substantial goodput improvements under strict service level objectives, although the actual gain depends heavily on prompt and output lengths and on interconnect speed.
Implementations and deployment choices
Disaggregated serving moved from research to production between 2023 and 2025. Inference engines such as vLLM and SGLang added prefill-decode separation, and orchestration frameworks such as NVIDIA Dynamo and llm-d coordinate routing and KV cache movement across clusters. Large model providers have described similar designs for their own fleets.
Choosing whether to disaggregate depends on workload shape and scale. The following decision table summarizes common situations.
| Situation | Recommended approach | Reason |
|---|---|---|
| Long prompts, short outputs | Disaggregate with more prefill workers | Prefill dominates and interferes with decode |
| Short prompts, long outputs | Colocated with continuous batching | Little prefill pressure to isolate |
| Strict TTFT and TPOT targets | Disaggregate | Each phase gets its own latency budget |
| Single node or few GPUs | Colocated with chunked prefill | Transfer overhead outweighs the benefit |
| Heavy prefix reuse in RAG or agents | Disaggregate with cache-aware routing | Shared KV cache reduces repeated prefill |
Some deployments go further and disaggregate attention from feed-forward layers or offload KV cache to CPU memory and storage tiers. These variants follow the same principle: place each part of inference on the resource that limits it.
Risks and common mistakes
The architecture introduces failure modes that colocated serving does not have. Most problems come from underestimating transfer cost or from static pool sizing that ignores changing traffic.
- Slow interconnects: moving the KV cache over standard TCP networking can erase the latency gains entirely.
- Fixed prefill-to-decode ratios: traffic shifts during the day, and an unbalanced ratio leaves one pool idle while the other queues.
- Ignoring prefix caching: routing without cache awareness repeats prefill work that could have been reused.
- Weak observability: latency must be traced across both pools and the transfer step to find bottlenecks.
- Premature adoption: small deployments gain little and pay the full complexity cost.
Stripped to the essentials
Answering a question with a language model happens in two steps. First the model reads the whole question at once, which needs heavy raw calculation. Then it writes the answer one word at a time, which mostly needs fast access to memory.
Disaggregated serving gives each step its own team of machines, like a restaurant where one kitchen preps ingredients and another finishes the dishes. Each team is set up for its own job, the prepared work is handed over quickly, and customers wait less, even when the restaurant is busy.
Where it is applied
A company runs an internal coding assistant on a 70-billion-parameter model. Developers paste prompts of 6,000 to 20,000 tokens that include source files, but most answers are only a few hundred tokens long. On colocated servers, large pasted files caused streamed answers to freeze for over a second whenever a new long prompt arrived.
The infrastructure team splits its 32 GPUs into 12 prefill and 20 decode workers, connected through InfiniBand, and adds a router that prefers workers holding a matching prefix cache. Prefill workers use higher tensor parallelism; decode workers run large continuous batches.
After the change, median time to first token falls noticeably and the generation pauses disappear, while the same hardware serves more concurrent users within the latency target. The team monitors the pool ratio weekly and shifts two GPUs to prefill during release weeks, when prompts grow longer.