Continuous batching, also called in-flight batching, is an inference scheduling technique in which a language model server adds and removes requests at every decoding step, instead of waiting for a whole batch to complete. In AI infrastructure, it is the main reason modern inference servers achieve several times higher throughput than naive serving on the same GPUs.
The technique exploits the token-by-token nature of autoregressive generation. Because each request produces output one token per iteration and stops at a different length, scheduling at the granularity of a single iteration keeps the accelerator busy with useful work rather than idle padding.
Why static batching wastes GPU capacity
Static batching groups a fixed set of requests, runs them together until the longest one finishes, and only then accepts new work. For large language models this is inefficient, because output lengths vary from a handful of tokens to thousands, so short requests sit idle while the batch waits for the longest generation.
Decoding is also memory-bandwidth bound: each step reads all model weights from GPU memory to produce a single token per sequence. Serving one request at a time leaves most compute units unused, while a larger concurrent batch amortizes the same weight reads across many sequences. Continuous batching keeps that batch full at every step.
The comparison below shows how the main batching strategies handle arrivals and completions.
| Strategy | When batch membership changes | Main drawback |
|---|---|---|
| No batching | Never; one request at a time | Very low GPU utilization |
| Static batching | Only after the whole batch finishes | Idle slots and long queue waits |
| Dynamic batching | When a batch is formed, after a short wait window | Still bound to the longest sequence |
| Continuous batching | At every decoding iteration | More complex scheduler and memory management |
How iteration-level scheduling works
Continuous batching relies on an iteration-level scheduler that reconsiders the batch before each forward pass. The idea was formalized in research on the Orca serving system, and it is now standard in engines such as vLLM, TensorRT-LLM, Text Generation Inference and SGLang.
- The scheduler checks the queue and the free KV cache memory.
- New requests that fit are admitted and their prompts enter the prefill phase.
- All active sequences run one decode step together and each emits one token.
- Sequences that hit an end-of-sequence token or a length limit are evicted immediately.
- Freed memory and batch slots become available for the next iteration.
Mixing prefill and decode is the hardest part. Prefill processes a whole prompt at once and is compute-bound, while decode is memory-bound, so a long prompt admitted in one step can stall token generation for everyone. Chunked prefill splits long prompts into slices that share iterations with ongoing decodes, which smooths latency at a small cost in prefill speed.
Memory management determines how many sequences fit. PagedAttention stores the KV cache in fixed-size blocks, similar to virtual memory pages, which removes fragmentation and lets the scheduler admit requests without reserving memory for the maximum possible output length.
Key parameters and their effects
Operators tune continuous batching with a small set of scheduler limits that balance throughput, latency and memory safety. The names differ between engines, but the underlying controls are consistent.
| Parameter | What it limits | Effect of raising it |
|---|---|---|
| Maximum concurrent sequences | Number of active requests per step | Higher throughput, slower per-token speed |
| Maximum batched tokens | Tokens processed in one iteration | Faster prefill, larger latency spikes |
| KV cache memory fraction | GPU memory reserved for attention state | More concurrency, less headroom |
| Prefill chunk size | Prompt tokens processed per iteration | Faster first token, more decode jitter |
| Preemption policy | What happens when memory runs out | Swap or recompute evicted sequences |
When the KV cache fills, the scheduler must preempt running sequences, either by swapping their cache to CPU memory or by discarding it and recomputing later. Frequent preemption is a clear signal that concurrency limits are set too high for the available memory.
Throughput and latency trade-offs
Continuous batching raises aggregate throughput in tokens per second, but it does not make every individual request faster. Each extra sequence in the batch adds work to every step, so time per output token rises as concurrency grows, while time to first token depends on queue depth and prefill scheduling.
| Metric | Low occupancy | High occupancy |
|---|---|---|
| Tokens per second (total) | Low; GPU underused | High; near hardware limit |
| Time per output token | Lowest | Noticeably higher |
| Time to first token | Short | Longer if prefill queues build up |
| Cost per token | High | Lowest |
Reported throughput gains over static batching range from two to more than twenty times, depending on output length variance, model size and memory management. Workloads with highly variable output lengths benefit most; uniform, short outputs gain less. Interactive products therefore cap concurrency to meet a latency target, while offline batch inference can run the batch as full as memory allows.
Limitations and common mistakes
Continuous batching is not free performance, and several misunderstandings appear repeatedly in production deployments. The most common ones relate to memory, fairness and measurement.
- Benchmarking with identical prompts hides real behavior; use realistic length distributions.
- Ignoring tail latency makes averages look fine while p99 time to first token degrades.
- Oversized context limits reserve memory that could serve more concurrent users.
- Assuming it replaces other optimizations; it combines with speculative decoding, quantization and prompt caching.
- Running one pool for chat and batch jobs lets long offline requests hurt interactive latency.
At very large scale, the conflict between prefill and decode can justify disaggregated serving, where separate GPU pools handle each phase. Continuous batching still operates inside each pool, so it remains the baseline scheduling layer of modern model serving.
Explained without jargon
Imagine a taxi minibus that waits until every passenger has reached their stop before picking anyone new up. Seats stay empty for most of the trip. Continuous batching works like a bus that lets people off and on at every stop. As soon as one passenger leaves, a waiting passenger takes the seat.
For an AI model, the passengers are user requests and the seats are slots on the graphics processor. Keeping every seat filled means the same hardware answers many more people. Each single ride may take a little longer, but the overall service is far cheaper.
A typical use case
A software company runs an internal coding assistant on four GPUs with a 70-billion-parameter model. With a simple server that processed fixed batches, the system handled about 12 concurrent developers before responses slowed to an unacceptable pace, because short answers waited for long code generations to finish.
The platform team migrated to an inference server with continuous batching, PagedAttention and chunked prefill, and set a concurrency limit that kept time per output token under 60 milliseconds. Under the same traffic pattern, the cluster served more than 80 concurrent developers with similar response times. Cost per generated token fell by roughly a factor of six, which postponed a planned hardware purchase by a full budget year.