Dynamic batching is a model serving technique in which an inference server collects individual requests as they arrive and combines them into a single batch at runtime, bounded by a maximum batch size and a maximum queue delay. In AI infrastructure, it lets one forward pass on a GPU or other accelerator serve many clients at once, which raises throughput and utilization without requiring clients to batch their own calls.
The technique trades a small, controlled amount of added latency for much better hardware efficiency. It is a standard feature of inference servers such as NVIDIA Triton Inference Server and TorchServe, and it differs from the iteration-level continuous batching used for autoregressive large language models.
How dynamic batching forms batches
Requests enter a scheduler queue instead of going straight to the model. The scheduler releases a batch when either of two conditions is met: the queue holds enough requests to reach the preferred or maximum batch size, or the oldest request has waited for the configured delay window, typically between a fraction of a millisecond and a few tens of milliseconds.
Once released, the requests are padded or concatenated into one tensor, executed in a single forward pass and split back into individual responses. Because accelerators process matrix operations in parallel, a batch of 16 inputs often takes only modestly longer than a batch of one, so the cost per request falls sharply as batches fill.
The table below lists the parameters that control this behavior and their typical effect.
| Parameter | Typical range | Effect of increasing it |
|---|---|---|
| Maximum batch size | 4 to 256 inputs | Higher peak throughput; more memory use |
| Maximum queue delay | 0.1 to 50 ms | Fuller batches; higher tail latency |
| Preferred batch sizes | Powers of two, such as 8 or 32 | Better kernel efficiency at those sizes |
| Model instance count | 1 to several per GPU | More concurrency; more memory pressure |
| Queue timeout or priority | Per request class | Protects latency-sensitive traffic |
The throughput and latency trade-off
Dynamic batching shifts the operating point of a serving system along a throughput-latency curve. Without batching, each request runs alone and the accelerator sits largely idle between kernel launches. With batching, the accelerator works on larger matrices, but each request may wait in the queue and then share compute time with its batch neighbors.
The benefit depends on traffic intensity. Under low load, few requests arrive within the delay window, so batches stay small and the added wait buys little. Under high load, batches fill almost instantly and the delay window rarely expires, so throughput gains are largest exactly when they are needed.
| Traffic level | Typical batch size | Latency impact | Throughput gain |
|---|---|---|---|
| Very low | 1 to 2 | Up to the full delay window | Negligible |
| Moderate | Partial batches | Small added wait | Noticeable |
| High | At or near maximum | Dominated by compute time | Largest |
| Overload | Maximum | Queue grows without bound | Saturated; needs scaling |
A practical rule is to set the delay window to a small fraction of the latency budget. If a service promises a 100 ms p99 response time and the model needs 40 ms per batch, a delay of a few milliseconds leaves headroom for queueing and network overhead.
Dynamic batching compared with other batching strategies
Dynamic batching sits between offline batch inference and the iteration-level scheduling used for generative models. Each strategy fits a different workload, and the neighboring Terms continuous batching and batch inference describe the other two in depth.
| Strategy | Batch formed | Best suited to |
|---|---|---|
| No batching | Never; one request per pass | Very low traffic, strict latency |
| Static batching | Fixed size set by the client or job | Offline scoring with known volume |
| Dynamic batching | At runtime, by size or time limit | Online classifiers, embeddings, vision models |
| Continuous batching | Per decoding step, requests join and leave | Autoregressive LLM text generation |
| Batch inference | Large offline jobs over stored data | Nightly scoring, dataset labeling |
The key distinction for large language models is request granularity. Classic dynamic batching holds a batch together until every member finishes, so a short completion waits for the longest one in its batch. Continuous batching removes that head-of-line blocking by admitting and retiring sequences at every token step, which is why engines such as vLLM rely on it together with PagedAttention and the KV cache.
Configuring dynamic batching in practice
Tuning starts with measuring the model alone at different batch sizes to find where throughput stops improving or memory runs out. The maximum batch size is then set near that knee, and the delay window is increased gradually while p95 and p99 latency are monitored under realistic load.
- Profile per batch size with a load generator to build a throughput and latency table before going live.
- Bucket variable-length inputs by sequence length to limit wasted computation on padding.
- Separate traffic classes so interactive requests are not delayed behind bulk jobs.
- Track batch-size distribution in monitoring; consistently tiny batches mean the delay window or instance count needs adjustment.
- Combine with autoscaling, because batching raises capacity per GPU but cannot absorb sustained overload.
Models with fixed input shapes, such as image classifiers and embedding encoders, benefit most because batching them adds no padding. Models with highly variable input lengths need bucketing or ragged-batch support to avoid spending compute on padding tokens.
Limitations and common mistakes
Dynamic batching is not free. It adds queueing delay, memory pressure and operational complexity, and badly chosen settings can make a service slower without improving cost. The following table lists frequent symptoms and their usual causes.
| Symptom | Likely cause | Fix |
|---|---|---|
| High latency at low traffic | Delay window too long | Shorten the window or disable batching |
| Out-of-memory errors | Maximum batch size too large | Lower the limit or use smaller precision |
| Low GPU utilization under load | Batches never fill | Raise delay or reduce model instances |
| Long tail latency for short inputs | Padding to the longest input | Bucket requests by length |
A common misconception is that larger batches always lower cost. Beyond the compute saturation point, throughput stays flat while latency keeps rising, so the optimal batch size must be measured for each model and accelerator, not copied from another deployment.
Stripped to the essentials
Dynamic batching works like a shuttle bus that waits a moment before leaving. Instead of driving one passenger at a time, it waits a few seconds to see if others arrive, then carries everyone together. Each passenger may wait slightly longer, but the bus makes far fewer trips.
An AI server does the same with requests. It gathers questions that arrive within a few milliseconds and sends them through the model together. The chip does much more useful work, so the same hardware serves many more users.
Where it is applied
An online retailer runs a text embedding model that converts search queries into vectors for product retrieval. At peak hours the service receives several thousand queries per second, and serving each query individually kept the GPUs busy only a small share of the time, so the team needed many expensive instances.
The engineers enabled dynamic batching on their inference server with a maximum batch size of 64 and a delay window of 2 ms. Load tests showed batches of 40 to 64 requests during peaks, while p99 latency stayed well inside the 50 ms budget. The result was several times higher throughput per GPU, which let the team cut the number of serving instances substantially while keeping search responses fast.