Stochastic gradient descent (SGD) is an optimization algorithm that updates model parameters using a gradient estimated from a small random sample of training data rather than from the entire dataset. In artificial intelligence, it is the foundation of almost every neural network training procedure, from small classifiers to large language models.
The word stochastic refers to the randomness of the sampled examples. Each step is cheap and noisy, but the noise averages out over many steps, so SGD reaches good solutions at a fraction of the computational cost of exact gradient descent.
How stochastic gradient descent updates parameters
SGD repeatedly moves the parameters a small step against the gradient of the loss function, computed on a randomly drawn mini-batch. The update rule is w = w - lr * g, where w is the parameter vector, lr is the learning rate and g is the mini-batch gradient obtained by backpropagation.
Because the mini-batch is a random sample, g is an unbiased estimate of the true gradient: on average it points in the same direction as the full-dataset gradient, but any single estimate carries sampling variance. That variance shrinks roughly in proportion to one over the batch size, which is why larger batches give smoother but more expensive steps.
The steps of one training iteration follow a fixed cycle that is repeated millions of times in a large training run.
- Shuffle the dataset or draw a random mini-batch of examples.
- Run a forward pass and compute the loss on that mini-batch.
- Run backpropagation to obtain the gradient of the loss for every parameter.
- Update each parameter by subtracting the learning rate times its gradient.
- Repeat until the epoch ends, then reshuffle and continue.
Batch, mini-batch and single-sample variants
The term SGD covers a family of methods that differ mainly in how many examples feed each gradient estimate. Strictly, classic SGD uses one example per step, but in modern practice the name almost always means mini-batch SGD. The table below compares the three regimes.
| Variant | Examples per step | Gradient noise | Typical use |
|---|---|---|---|
| Full-batch gradient descent | Entire dataset | None | Small convex problems |
| Single-sample SGD | 1 | Very high | Online learning, streaming data |
| Mini-batch SGD | Typically 32 to several thousand | Moderate | Deep learning, almost all modern training |
Mini-batches dominate because they match the parallel hardware of GPUs and TPUs. A batch of 256 examples costs little more wall-clock time than a batch of 32 on a modern accelerator, while one example per step leaves most of the chip idle.
Key hyperparameters of SGD
The behavior of SGD depends on a small set of hyperparameters that interact closely. Changing one usually requires retuning another, most notably the pairing of learning rate and batch size.
| Hyperparameter | Common range | What it controls |
|---|---|---|
| Learning rate | 0.001 to 0.5 for plain SGD | Step length; too high diverges, too low stalls |
| Batch size | 32 to 65,536 examples | Gradient noise and hardware efficiency |
| Momentum | 0.9 is the usual default | Smoothing of updates across steps |
| Weight decay | 0.00001 to 0.0005 | Penalty that keeps weights small |
| Learning rate schedule | Step, cosine or linear decay | How the step size shrinks over training |
A widely used heuristic is the linear scaling rule: when the batch size is multiplied by k, the learning rate is multiplied by k as well, usually combined with a short warmup period at the start of training. The rule holds well up to a critical batch size, beyond which larger batches give diminishing returns per example processed.
SGD compared with related optimizers
SGD is the baseline against which other optimizers are measured, and most popular optimizers are modifications of it. They keep the stochastic mini-batch gradient but change how that gradient becomes an update.
| Optimizer | Change relative to plain SGD | Strength | Weakness |
|---|---|---|---|
| SGD with momentum | Accumulates a velocity of past gradients | Faster progress in narrow valleys | One more hyperparameter |
| Nesterov momentum | Evaluates gradient at a look-ahead point | Slightly more stable convergence | Marginal gain in deep networks |
| Adam optimizer | Per-parameter adaptive step sizes | Little tuning, robust for transformers | Higher memory use per parameter |
| AdamW | Adam with decoupled weight decay | Standard for large language models | Two extra state values per parameter |
In computer vision, SGD with momentum often matches or slightly beats adaptive methods on final accuracy for convolutional networks, which is why many image classification recipes still use it. For transformer models, Adam and AdamW are the default because plain SGD tends to train them slowly and unstably.
Why gradient noise can help
The noise in SGD is not only a cost; it also acts as implicit regularization. Random fluctuations help the optimizer escape saddle points and sharp minima, which are common in the high-dimensional loss surfaces of deep networks.
Several studies associate the flatter minima that noisy SGD tends to find with better generalization to unseen data, although the link between flatness and generalization remains an open research question. What is well established in practice is that very large batches without adjusted schedules often reach lower training loss but worse test accuracy.
Common problems and how to address them
Most SGD failures show up as recognizable patterns in the loss curve, and each has a standard remedy. The table maps symptoms to causes and fixes.
| Symptom | Likely cause | Typical fix |
|---|---|---|
| Loss becomes NaN or explodes | Learning rate too high | Lower the rate, add gradient clipping |
| Loss decreases extremely slowly | Learning rate too low | Raise the rate or add momentum |
| Loss oscillates without settling | No decay in the schedule | Use step or cosine decay |
| Training loss low, validation loss rising | Overfitting | Add weight decay or early stopping |
| Sudden loss spikes late in training | Unstable batches or rate | Clip gradients, reduce peak rate |
Two further mistakes are frequent. Failing to shuffle the data between epochs introduces ordering bias, and changing the batch size without retuning the learning rate silently alters the effective step size.
Put simply
Imagine walking downhill in thick fog to reach the lowest point of a valley. Measuring the slope of the entire valley before every step would take forever. Instead, you check the ground around your feet, take a small step downhill, and repeat.
Stochastic gradient descent works the same way. Instead of studying all the training data before each change, the model looks at a small random handful of examples, adjusts itself slightly, and moves on. Each step is a little imprecise, but thousands of quick, rough steps beat a few slow, perfect ones.
In practice
A machine learning team trains a convolutional network to classify 1.2 million product photos into 500 categories. Computing the exact gradient over all images for each update would take minutes per step on their eight-GPU server, so they use mini-batch SGD with a batch of 512 images, momentum of 0.9 and weight decay of 0.0001.
They start with a learning rate of 0.2, warm it up over the first five epochs, and then apply cosine decay over 90 epochs. After one early run diverges at a rate of 0.8, they add gradient clipping and return to the planned value. The final model reaches its target validation accuracy in about two days, a run that would have been impractical with full-batch gradient descent.