Batch size is the number of training examples processed together before a model computes one gradient estimate and performs one optimizer step. Training in artificial intelligence rarely uses the whole dataset per update, so batch size defines how many samples, or in language models how many tokens, contribute to each step of stochastic gradient descent and its variants such as the Adam optimizer.
The choice looks like a simple hardware setting, yet it couples statistical behavior, memory consumption and wall-clock time. It interacts tightly with the learning rate, the learning rate schedule and the total training budget, which is why practitioners treat it as one of the most consequential hyperparameters in any training run.
How batch size shapes each gradient step
Each optimizer step uses the average gradient over the batch as an estimate of the true gradient of the loss function across the full dataset. A larger batch averages more samples, so the estimate has lower variance; the standard error of the mean gradient shrinks roughly with the square root of the batch size.
That noise is not purely harmful. Gradient noise acts as implicit regularization, helping the optimizer escape sharp regions of the loss surface and often improving generalization. Very large batches remove much of this noise, which can make training more deterministic but sometimes leads to sharper minima and weaker test performance unless other settings compensate.
The table below summarizes how the main regimes of batch size behave during training.
| Regime | Samples per step | Gradient noise | Typical consequence |
|---|---|---|---|
| Stochastic (online) | 1 | Very high | Slow on hardware, erratic loss curve |
| Mini-batch | 16 to 512 | Moderate | Default balance for most vision and tabular models |
| Large batch | Thousands to millions of tokens | Low | Efficient on clusters, needs tuned learning rate |
| Full batch | Entire dataset | None | Deterministic, rarely practical for large data |
Batch size, learning rate and the critical batch size
Batch size and learning rate must be tuned together because both control the effective step size of training. A widely used heuristic, the linear scaling rule, multiplies the learning rate by the same factor as the batch size, usually combined with a learning rate warmup to avoid early instability. For adaptive optimizers such as Adam, a square-root scaling rule often fits better.
Scaling the batch up does not help indefinitely. Empirical work on the critical batch size, often estimated through the gradient noise scale, shows that below a threshold doubling the batch roughly halves the number of steps needed, while above it the returns diminish sharply. The critical batch size tends to grow as the loss falls, which explains why large language model pretraining often uses a batch size warmup that increases the batch during training.
The next table shows how common scaling adjustments relate to batch size changes.
| Change | Recommended adjustment | Reason |
|---|---|---|
| Double batch size with SGD | Roughly double the learning rate | Keeps the update magnitude per epoch similar |
| Double batch size with Adam | Increase learning rate by about 1.4x | Adaptive scaling responds more gently |
| Move to very large batches | Add warmup and consider LARS or LAMB | Layer-wise scaling stabilizes early steps |
| Reduce batch size sharply | Lower the learning rate | Noisier gradients amplify large steps |
Memory, hardware and how large batches are achieved
The batch size that fits on an accelerator is limited mainly by activation memory, which grows linearly with the number of samples and, for transformers, with the sequence length. Weights and optimizer states do not depend on batch size, so the marginal memory cost of each extra sample is the activations it stores for the backward pass.
Practitioners therefore distinguish several related quantities. The global batch size is what the optimizer sees, while each device processes a smaller micro-batch. Techniques such as gradient accumulation, gradient checkpointing and mixed-precision training let teams reach a target global batch without buying more memory.
| Term | Definition | Relationship |
|---|---|---|
| Micro-batch | Samples processed in one forward and backward pass on one device | Limited by device memory |
| Accumulation steps | Micro-batches summed before an update | Multiplies the micro-batch |
| Data-parallel degree | Number of model replicas processing different data | Multiplies the per-device batch |
| Global batch size | Samples contributing to one optimizer step | Micro-batch x accumulation x replicas |
| Tokens per batch | Global batch size x sequence length | Standard unit in language model pretraining |
Batch normalization adds a further constraint: its statistics are computed per micro-batch, so very small per-device batches below about 8 to 16 samples degrade it. Group normalization, layer normalization or synchronized batch normalization avoid this dependency.
Choosing a batch size in practice
A sound choice starts from the largest batch that fits comfortably in memory, then checks whether it lies below the critical batch size for the task. Powers of two are conventional because they align with accelerator kernels, although the benefit is usually small on modern hardware.
| Situation | Typical batch size | Why |
|---|---|---|
| Small dataset, limited GPU | 16 to 64 | Fits memory and adds useful regularizing noise |
| Image classification at scale | 256 to 8,192 | Linear scaling with warmup preserves accuracy |
| LLM pretraining | About 1 to 16 million tokens | Matches the high critical batch size of late training |
| Fine-tuning or LoRA | 8 to 128 sequences | Small datasets reward more frequent updates |
| Contrastive learning | Thousands of pairs | More in-batch negatives improve the signal |
Reporting matters as much as choosing. Because the same number of epochs at different batch sizes yields a different number of optimizer steps, comparisons between runs should state the global batch size, sequence length, step count and learning rate together.
Common mistakes and misconceptions
Several recurring errors come from treating batch size as an isolated setting rather than part of the optimization recipe.
- Changing batch size without retuning the learning rate, which silently alters the effective step size and can cause divergence or slow convergence.
- Assuming bigger batches always train faster; beyond the critical batch size, extra samples buy almost no reduction in steps.
- Confusing the per-device batch with the global batch when reading papers or reproducing results.
- Ignoring that gradient accumulation does not reproduce batch normalization statistics of a true large batch.
- Blaming a loss spike on data when an aggressive batch-to-learning-rate ratio is the cause.
Explained without jargon
Batch size is how many examples a model looks at before it adjusts itself once. Imagine a teacher grading essays to decide how to change a lesson. Reading one essay gives a shaky impression; reading fifty gives a much steadier picture, but takes longer before any change is made.
Small batches mean frequent, rough corrections; large batches mean fewer, steadier corrections. The best choice depends on the computer memory available and on how quickly and reliably the model needs to learn.
A typical use case
A machine learning team fine-tunes an image classifier on 40,000 labeled product photos using a single GPU with 24 GB of memory. At a batch size of 256, training crashes with an out-of-memory error, while a batch size of 64 fits with room to spare.
To match the behavior of a published recipe that used 256, the engineers keep a micro-batch of 64 with four accumulation steps and keep the original learning rate of 0.1 with a five-epoch warmup. They switch from batch normalization to group normalization because the small micro-batch made the statistics unstable.
The result is a training curve that closely follows the reference run, reaching comparable validation accuracy within one percentage point, without any additional hardware and with only a modest increase in wall-clock time.