Batch size

On this page 7

Batch size is the number of training examples processed together before a model computes one gradient estimate and performs one optimizer step. Training in artificial intelligence rarely uses the whole dataset per update, so batch size defines how many samples, or in language models how many tokens, contribute to each step of stochastic gradient descent and its variants such as the Adam optimizer.

The choice looks like a simple hardware setting, yet it couples statistical behavior, memory consumption and wall-clock time. It interacts tightly with the learning rate, the learning rate schedule and the total training budget, which is why practitioners treat it as one of the most consequential hyperparameters in any training run.

How batch size shapes each gradient step

Each optimizer step uses the average gradient over the batch as an estimate of the true gradient of the loss function across the full dataset. A larger batch averages more samples, so the estimate has lower variance; the standard error of the mean gradient shrinks roughly with the square root of the batch size.

That noise is not purely harmful. Gradient noise acts as implicit regularization, helping the optimizer escape sharp regions of the loss surface and often improving generalization. Very large batches remove much of this noise, which can make training more deterministic but sometimes leads to sharper minima and weaker test performance unless other settings compensate.

The table below summarizes how the main regimes of batch size behave during training.

RegimeSamples per stepGradient noiseTypical consequence
Stochastic (online)1Very highSlow on hardware, erratic loss curve
Mini-batch16 to 512ModerateDefault balance for most vision and tabular models
Large batchThousands to millions of tokensLowEfficient on clusters, needs tuned learning rate
Full batchEntire datasetNoneDeterministic, rarely practical for large data
Batch size regimes and their typical behavior

Batch size, learning rate and the critical batch size

Batch size and learning rate must be tuned together because both control the effective step size of training. A widely used heuristic, the linear scaling rule, multiplies the learning rate by the same factor as the batch size, usually combined with a learning rate warmup to avoid early instability. For adaptive optimizers such as Adam, a square-root scaling rule often fits better.

Scaling the batch up does not help indefinitely. Empirical work on the critical batch size, often estimated through the gradient noise scale, shows that below a threshold doubling the batch roughly halves the number of steps needed, while above it the returns diminish sharply. The critical batch size tends to grow as the loss falls, which explains why large language model pretraining often uses a batch size warmup that increases the batch during training.

The next table shows how common scaling adjustments relate to batch size changes.

ChangeRecommended adjustmentReason
Double batch size with SGDRoughly double the learning rateKeeps the update magnitude per epoch similar
Double batch size with AdamIncrease learning rate by about 1.4xAdaptive scaling responds more gently
Move to very large batchesAdd warmup and consider LARS or LAMBLayer-wise scaling stabilizes early steps
Reduce batch size sharplyLower the learning rateNoisier gradients amplify large steps
Adjusting related hyperparameters when batch size changes

Memory, hardware and how large batches are achieved

The batch size that fits on an accelerator is limited mainly by activation memory, which grows linearly with the number of samples and, for transformers, with the sequence length. Weights and optimizer states do not depend on batch size, so the marginal memory cost of each extra sample is the activations it stores for the backward pass.

Practitioners therefore distinguish several related quantities. The global batch size is what the optimizer sees, while each device processes a smaller micro-batch. Techniques such as gradient accumulation, gradient checkpointing and mixed-precision training let teams reach a target global batch without buying more memory.

TermDefinitionRelationship
Micro-batchSamples processed in one forward and backward pass on one deviceLimited by device memory
Accumulation stepsMicro-batches summed before an updateMultiplies the micro-batch
Data-parallel degreeNumber of model replicas processing different dataMultiplies the per-device batch
Global batch sizeSamples contributing to one optimizer stepMicro-batch x accumulation x replicas
Tokens per batchGlobal batch size x sequence lengthStandard unit in language model pretraining
Terms used to describe batch size in distributed training

Batch normalization adds a further constraint: its statistics are computed per micro-batch, so very small per-device batches below about 8 to 16 samples degrade it. Group normalization, layer normalization or synchronized batch normalization avoid this dependency.

Choosing a batch size in practice

A sound choice starts from the largest batch that fits comfortably in memory, then checks whether it lies below the critical batch size for the task. Powers of two are conventional because they align with accelerator kernels, although the benefit is usually small on modern hardware.

SituationTypical batch sizeWhy
Small dataset, limited GPU16 to 64Fits memory and adds useful regularizing noise
Image classification at scale256 to 8,192Linear scaling with warmup preserves accuracy
LLM pretrainingAbout 1 to 16 million tokensMatches the high critical batch size of late training
Fine-tuning or LoRA8 to 128 sequencesSmall datasets reward more frequent updates
Contrastive learningThousands of pairsMore in-batch negatives improve the signal
Decision guide for common training situations

Reporting matters as much as choosing. Because the same number of epochs at different batch sizes yields a different number of optimizer steps, comparisons between runs should state the global batch size, sequence length, step count and learning rate together.

Common mistakes and misconceptions

Several recurring errors come from treating batch size as an isolated setting rather than part of the optimization recipe.

  • Changing batch size without retuning the learning rate, which silently alters the effective step size and can cause divergence or slow convergence.
  • Assuming bigger batches always train faster; beyond the critical batch size, extra samples buy almost no reduction in steps.
  • Confusing the per-device batch with the global batch when reading papers or reproducing results.
  • Ignoring that gradient accumulation does not reproduce batch normalization statistics of a true large batch.
  • Blaming a loss spike on data when an aggressive batch-to-learning-rate ratio is the cause.

Explained without jargon

Batch size is how many examples a model looks at before it adjusts itself once. Imagine a teacher grading essays to decide how to change a lesson. Reading one essay gives a shaky impression; reading fifty gives a much steadier picture, but takes longer before any change is made.

Small batches mean frequent, rough corrections; large batches mean fewer, steadier corrections. The best choice depends on the computer memory available and on how quickly and reliably the model needs to learn.

A typical use case

A machine learning team fine-tunes an image classifier on 40,000 labeled product photos using a single GPU with 24 GB of memory. At a batch size of 256, training crashes with an out-of-memory error, while a batch size of 64 fits with room to spare.

To match the behavior of a published recipe that used 256, the engineers keep a micro-batch of 64 with four accumulation steps and keep the original learning rate of 0.1 with a five-epoch warmup. They switch from batch normalization to group normalization because the small micro-batch made the statistics unstable.

The result is a training curve that closely follows the reference run, reaching comparable validation accuracy within one percentage point, without any additional hardware and with only a modest increase in wall-clock time.