Stochastic gradient descent

Also known as: SGD

On this page 8

Stochastic gradient descent (SGD) is an optimization algorithm that updates model parameters using a gradient estimated from a small random sample of training data rather than from the entire dataset. In artificial intelligence, it is the foundation of almost every neural network training procedure, from small classifiers to large language models.

The word stochastic refers to the randomness of the sampled examples. Each step is cheap and noisy, but the noise averages out over many steps, so SGD reaches good solutions at a fraction of the computational cost of exact gradient descent.

How stochastic gradient descent updates parameters

SGD repeatedly moves the parameters a small step against the gradient of the loss function, computed on a randomly drawn mini-batch. The update rule is w = w - lr * g, where w is the parameter vector, lr is the learning rate and g is the mini-batch gradient obtained by backpropagation.

Because the mini-batch is a random sample, g is an unbiased estimate of the true gradient: on average it points in the same direction as the full-dataset gradient, but any single estimate carries sampling variance. That variance shrinks roughly in proportion to one over the batch size, which is why larger batches give smoother but more expensive steps.

The steps of one training iteration follow a fixed cycle that is repeated millions of times in a large training run.

  1. Shuffle the dataset or draw a random mini-batch of examples.
  2. Run a forward pass and compute the loss on that mini-batch.
  3. Run backpropagation to obtain the gradient of the loss for every parameter.
  4. Update each parameter by subtracting the learning rate times its gradient.
  5. Repeat until the epoch ends, then reshuffle and continue.

Batch, mini-batch and single-sample variants

The term SGD covers a family of methods that differ mainly in how many examples feed each gradient estimate. Strictly, classic SGD uses one example per step, but in modern practice the name almost always means mini-batch SGD. The table below compares the three regimes.

VariantExamples per stepGradient noiseTypical use
Full-batch gradient descentEntire datasetNoneSmall convex problems
Single-sample SGD1Very highOnline learning, streaming data
Mini-batch SGDTypically 32 to several thousandModerateDeep learning, almost all modern training
Gradient descent variants by sample size

Mini-batches dominate because they match the parallel hardware of GPUs and TPUs. A batch of 256 examples costs little more wall-clock time than a batch of 32 on a modern accelerator, while one example per step leaves most of the chip idle.

Key hyperparameters of SGD

The behavior of SGD depends on a small set of hyperparameters that interact closely. Changing one usually requires retuning another, most notably the pairing of learning rate and batch size.

HyperparameterCommon rangeWhat it controls
Learning rate0.001 to 0.5 for plain SGDStep length; too high diverges, too low stalls
Batch size32 to 65,536 examplesGradient noise and hardware efficiency
Momentum0.9 is the usual defaultSmoothing of updates across steps
Weight decay0.00001 to 0.0005Penalty that keeps weights small
Learning rate scheduleStep, cosine or linear decayHow the step size shrinks over training
Main SGD hyperparameters and their effects

A widely used heuristic is the linear scaling rule: when the batch size is multiplied by k, the learning rate is multiplied by k as well, usually combined with a short warmup period at the start of training. The rule holds well up to a critical batch size, beyond which larger batches give diminishing returns per example processed.

SGD is the baseline against which other optimizers are measured, and most popular optimizers are modifications of it. They keep the stochastic mini-batch gradient but change how that gradient becomes an update.

OptimizerChange relative to plain SGDStrengthWeakness
SGD with momentumAccumulates a velocity of past gradientsFaster progress in narrow valleysOne more hyperparameter
Nesterov momentumEvaluates gradient at a look-ahead pointSlightly more stable convergenceMarginal gain in deep networks
Adam optimizerPer-parameter adaptive step sizesLittle tuning, robust for transformersHigher memory use per parameter
AdamWAdam with decoupled weight decayStandard for large language modelsTwo extra state values per parameter
SGD and its main successors

In computer vision, SGD with momentum often matches or slightly beats adaptive methods on final accuracy for convolutional networks, which is why many image classification recipes still use it. For transformer models, Adam and AdamW are the default because plain SGD tends to train them slowly and unstably.

Why gradient noise can help

The noise in SGD is not only a cost; it also acts as implicit regularization. Random fluctuations help the optimizer escape saddle points and sharp minima, which are common in the high-dimensional loss surfaces of deep networks.

Several studies associate the flatter minima that noisy SGD tends to find with better generalization to unseen data, although the link between flatness and generalization remains an open research question. What is well established in practice is that very large batches without adjusted schedules often reach lower training loss but worse test accuracy.

Common problems and how to address them

Most SGD failures show up as recognizable patterns in the loss curve, and each has a standard remedy. The table maps symptoms to causes and fixes.

SymptomLikely causeTypical fix
Loss becomes NaN or explodesLearning rate too highLower the rate, add gradient clipping
Loss decreases extremely slowlyLearning rate too lowRaise the rate or add momentum
Loss oscillates without settlingNo decay in the scheduleUse step or cosine decay
Training loss low, validation loss risingOverfittingAdd weight decay or early stopping
Sudden loss spikes late in trainingUnstable batches or rateClip gradients, reduce peak rate
Typical SGD training failures

Two further mistakes are frequent. Failing to shuffle the data between epochs introduces ordering bias, and changing the batch size without retuning the learning rate silently alters the effective step size.

Put simply

Imagine walking downhill in thick fog to reach the lowest point of a valley. Measuring the slope of the entire valley before every step would take forever. Instead, you check the ground around your feet, take a small step downhill, and repeat.

Stochastic gradient descent works the same way. Instead of studying all the training data before each change, the model looks at a small random handful of examples, adjusts itself slightly, and moves on. Each step is a little imprecise, but thousands of quick, rough steps beat a few slow, perfect ones.

In practice

A machine learning team trains a convolutional network to classify 1.2 million product photos into 500 categories. Computing the exact gradient over all images for each update would take minutes per step on their eight-GPU server, so they use mini-batch SGD with a batch of 512 images, momentum of 0.9 and weight decay of 0.0001.

They start with a learning rate of 0.2, warm it up over the first five epochs, and then apply cosine decay over 90 epochs. After one early run diverges at a rate of 0.8, they add gradient clipping and return to the planned value. The final model reaches its target validation accuracy in about two days, a run that would have been impractical with full-batch gradient descent.