Continuous batching

Also known as: in-flight batching

On this page 7

Continuous batching, also called in-flight batching, is an inference scheduling technique in which a language model server adds and removes requests at every decoding step, instead of waiting for a whole batch to complete. In AI infrastructure, it is the main reason modern inference servers achieve several times higher throughput than naive serving on the same GPUs.

The technique exploits the token-by-token nature of autoregressive generation. Because each request produces output one token per iteration and stops at a different length, scheduling at the granularity of a single iteration keeps the accelerator busy with useful work rather than idle padding.

Why static batching wastes GPU capacity

Static batching groups a fixed set of requests, runs them together until the longest one finishes, and only then accepts new work. For large language models this is inefficient, because output lengths vary from a handful of tokens to thousands, so short requests sit idle while the batch waits for the longest generation.

Decoding is also memory-bandwidth bound: each step reads all model weights from GPU memory to produce a single token per sequence. Serving one request at a time leaves most compute units unused, while a larger concurrent batch amortizes the same weight reads across many sequences. Continuous batching keeps that batch full at every step.

The comparison below shows how the main batching strategies handle arrivals and completions.

StrategyWhen batch membership changesMain drawback
No batchingNever; one request at a timeVery low GPU utilization
Static batchingOnly after the whole batch finishesIdle slots and long queue waits
Dynamic batchingWhen a batch is formed, after a short wait windowStill bound to the longest sequence
Continuous batchingAt every decoding iterationMore complex scheduler and memory management
Batching strategies for model inference

How iteration-level scheduling works

Continuous batching relies on an iteration-level scheduler that reconsiders the batch before each forward pass. The idea was formalized in research on the Orca serving system, and it is now standard in engines such as vLLM, TensorRT-LLM, Text Generation Inference and SGLang.

  1. The scheduler checks the queue and the free KV cache memory.
  2. New requests that fit are admitted and their prompts enter the prefill phase.
  3. All active sequences run one decode step together and each emits one token.
  4. Sequences that hit an end-of-sequence token or a length limit are evicted immediately.
  5. Freed memory and batch slots become available for the next iteration.

Mixing prefill and decode is the hardest part. Prefill processes a whole prompt at once and is compute-bound, while decode is memory-bound, so a long prompt admitted in one step can stall token generation for everyone. Chunked prefill splits long prompts into slices that share iterations with ongoing decodes, which smooths latency at a small cost in prefill speed.

Memory management determines how many sequences fit. PagedAttention stores the KV cache in fixed-size blocks, similar to virtual memory pages, which removes fragmentation and lets the scheduler admit requests without reserving memory for the maximum possible output length.

Key parameters and their effects

Operators tune continuous batching with a small set of scheduler limits that balance throughput, latency and memory safety. The names differ between engines, but the underlying controls are consistent.

ParameterWhat it limitsEffect of raising it
Maximum concurrent sequencesNumber of active requests per stepHigher throughput, slower per-token speed
Maximum batched tokensTokens processed in one iterationFaster prefill, larger latency spikes
KV cache memory fractionGPU memory reserved for attention stateMore concurrency, less headroom
Prefill chunk sizePrompt tokens processed per iterationFaster first token, more decode jitter
Preemption policyWhat happens when memory runs outSwap or recompute evicted sequences
Scheduler parameters in continuous batching

When the KV cache fills, the scheduler must preempt running sequences, either by swapping their cache to CPU memory or by discarding it and recomputing later. Frequent preemption is a clear signal that concurrency limits are set too high for the available memory.

Throughput and latency trade-offs

Continuous batching raises aggregate throughput in tokens per second, but it does not make every individual request faster. Each extra sequence in the batch adds work to every step, so time per output token rises as concurrency grows, while time to first token depends on queue depth and prefill scheduling.

MetricLow occupancyHigh occupancy
Tokens per second (total)Low; GPU underusedHigh; near hardware limit
Time per output tokenLowestNoticeably higher
Time to first tokenShortLonger if prefill queues build up
Cost per tokenHighLowest
Metric behavior as batch occupancy grows

Reported throughput gains over static batching range from two to more than twenty times, depending on output length variance, model size and memory management. Workloads with highly variable output lengths benefit most; uniform, short outputs gain less. Interactive products therefore cap concurrency to meet a latency target, while offline batch inference can run the batch as full as memory allows.

Limitations and common mistakes

Continuous batching is not free performance, and several misunderstandings appear repeatedly in production deployments. The most common ones relate to memory, fairness and measurement.

  • Benchmarking with identical prompts hides real behavior; use realistic length distributions.
  • Ignoring tail latency makes averages look fine while p99 time to first token degrades.
  • Oversized context limits reserve memory that could serve more concurrent users.
  • Assuming it replaces other optimizations; it combines with speculative decoding, quantization and prompt caching.
  • Running one pool for chat and batch jobs lets long offline requests hurt interactive latency.

At very large scale, the conflict between prefill and decode can justify disaggregated serving, where separate GPU pools handle each phase. Continuous batching still operates inside each pool, so it remains the baseline scheduling layer of modern model serving.

Explained without jargon

Imagine a taxi minibus that waits until every passenger has reached their stop before picking anyone new up. Seats stay empty for most of the trip. Continuous batching works like a bus that lets people off and on at every stop. As soon as one passenger leaves, a waiting passenger takes the seat.

For an AI model, the passengers are user requests and the seats are slots on the graphics processor. Keeping every seat filled means the same hardware answers many more people. Each single ride may take a little longer, but the overall service is far cheaper.

A typical use case

A software company runs an internal coding assistant on four GPUs with a 70-billion-parameter model. With a simple server that processed fixed batches, the system handled about 12 concurrent developers before responses slowed to an unacceptable pace, because short answers waited for long code generations to finish.

The platform team migrated to an inference server with continuous batching, PagedAttention and chunked prefill, and set a concurrency limit that kept time per output token under 60 milliseconds. Under the same traffic pattern, the cluster served more than 80 concurrent developers with similar response times. Cost per generated token fell by roughly a factor of six, which postponed a planned hardware purchase by a full budget year.