Dynamic batching

On this page 7

Dynamic batching is a model serving technique in which an inference server collects individual requests as they arrive and combines them into a single batch at runtime, bounded by a maximum batch size and a maximum queue delay. In AI infrastructure, it lets one forward pass on a GPU or other accelerator serve many clients at once, which raises throughput and utilization without requiring clients to batch their own calls.

The technique trades a small, controlled amount of added latency for much better hardware efficiency. It is a standard feature of inference servers such as NVIDIA Triton Inference Server and TorchServe, and it differs from the iteration-level continuous batching used for autoregressive large language models.

How dynamic batching forms batches

Requests enter a scheduler queue instead of going straight to the model. The scheduler releases a batch when either of two conditions is met: the queue holds enough requests to reach the preferred or maximum batch size, or the oldest request has waited for the configured delay window, typically between a fraction of a millisecond and a few tens of milliseconds.

Once released, the requests are padded or concatenated into one tensor, executed in a single forward pass and split back into individual responses. Because accelerators process matrix operations in parallel, a batch of 16 inputs often takes only modestly longer than a batch of one, so the cost per request falls sharply as batches fill.

The table below lists the parameters that control this behavior and their typical effect.

ParameterTypical rangeEffect of increasing it
Maximum batch size4 to 256 inputsHigher peak throughput; more memory use
Maximum queue delay0.1 to 50 msFuller batches; higher tail latency
Preferred batch sizesPowers of two, such as 8 or 32Better kernel efficiency at those sizes
Model instance count1 to several per GPUMore concurrency; more memory pressure
Queue timeout or priorityPer request classProtects latency-sensitive traffic
Key dynamic batching parameters

The throughput and latency trade-off

Dynamic batching shifts the operating point of a serving system along a throughput-latency curve. Without batching, each request runs alone and the accelerator sits largely idle between kernel launches. With batching, the accelerator works on larger matrices, but each request may wait in the queue and then share compute time with its batch neighbors.

The benefit depends on traffic intensity. Under low load, few requests arrive within the delay window, so batches stay small and the added wait buys little. Under high load, batches fill almost instantly and the delay window rarely expires, so throughput gains are largest exactly when they are needed.

Traffic levelTypical batch sizeLatency impactThroughput gain
Very low1 to 2Up to the full delay windowNegligible
ModeratePartial batchesSmall added waitNoticeable
HighAt or near maximumDominated by compute timeLargest
OverloadMaximumQueue grows without boundSaturated; needs scaling
Effect of load on dynamic batching

A practical rule is to set the delay window to a small fraction of the latency budget. If a service promises a 100 ms p99 response time and the model needs 40 ms per batch, a delay of a few milliseconds leaves headroom for queueing and network overhead.

Dynamic batching compared with other batching strategies

Dynamic batching sits between offline batch inference and the iteration-level scheduling used for generative models. Each strategy fits a different workload, and the neighboring Terms continuous batching and batch inference describe the other two in depth.

StrategyBatch formedBest suited to
No batchingNever; one request per passVery low traffic, strict latency
Static batchingFixed size set by the client or jobOffline scoring with known volume
Dynamic batchingAt runtime, by size or time limitOnline classifiers, embeddings, vision models
Continuous batchingPer decoding step, requests join and leaveAutoregressive LLM text generation
Batch inferenceLarge offline jobs over stored dataNightly scoring, dataset labeling
Batching strategies in model serving

The key distinction for large language models is request granularity. Classic dynamic batching holds a batch together until every member finishes, so a short completion waits for the longest one in its batch. Continuous batching removes that head-of-line blocking by admitting and retiring sequences at every token step, which is why engines such as vLLM rely on it together with PagedAttention and the KV cache.

Configuring dynamic batching in practice

Tuning starts with measuring the model alone at different batch sizes to find where throughput stops improving or memory runs out. The maximum batch size is then set near that knee, and the delay window is increased gradually while p95 and p99 latency are monitored under realistic load.

  • Profile per batch size with a load generator to build a throughput and latency table before going live.
  • Bucket variable-length inputs by sequence length to limit wasted computation on padding.
  • Separate traffic classes so interactive requests are not delayed behind bulk jobs.
  • Track batch-size distribution in monitoring; consistently tiny batches mean the delay window or instance count needs adjustment.
  • Combine with autoscaling, because batching raises capacity per GPU but cannot absorb sustained overload.

Models with fixed input shapes, such as image classifiers and embedding encoders, benefit most because batching them adds no padding. Models with highly variable input lengths need bucketing or ragged-batch support to avoid spending compute on padding tokens.

Limitations and common mistakes

Dynamic batching is not free. It adds queueing delay, memory pressure and operational complexity, and badly chosen settings can make a service slower without improving cost. The following table lists frequent symptoms and their usual causes.

SymptomLikely causeFix
High latency at low trafficDelay window too longShorten the window or disable batching
Out-of-memory errorsMaximum batch size too largeLower the limit or use smaller precision
Low GPU utilization under loadBatches never fillRaise delay or reduce model instances
Long tail latency for short inputsPadding to the longest inputBucket requests by length
Common symptoms and fixes

A common misconception is that larger batches always lower cost. Beyond the compute saturation point, throughput stays flat while latency keeps rising, so the optimal batch size must be measured for each model and accelerator, not copied from another deployment.

Stripped to the essentials

Dynamic batching works like a shuttle bus that waits a moment before leaving. Instead of driving one passenger at a time, it waits a few seconds to see if others arrive, then carries everyone together. Each passenger may wait slightly longer, but the bus makes far fewer trips.

An AI server does the same with requests. It gathers questions that arrive within a few milliseconds and sends them through the model together. The chip does much more useful work, so the same hardware serves many more users.

Where it is applied

An online retailer runs a text embedding model that converts search queries into vectors for product retrieval. At peak hours the service receives several thousand queries per second, and serving each query individually kept the GPUs busy only a small share of the time, so the team needed many expensive instances.

The engineers enabled dynamic batching on their inference server with a maximum batch size of 64 and a delay window of 2 ms. Load tests showed batches of 40 to 64 requests during peaks, while p99 latency stayed well inside the 50 ms budget. The result was several times higher throughput per GPU, which let the team cut the number of serving instances substantially while keeping search responses fast.