Gradient descent is the iterative optimization algorithm that minimizes a loss function by repeatedly stepping parameters against the gradient. In the training of artificial intelligence models, it is the engine that turns data into learned weights: from linear regression to large language models, almost every neural network is trained by some variant of gradient descent.
The idea is simple. The gradient points toward the steepest increase of the loss, so a small step against it reduces the loss. Repeated over thousands to millions of steps, these updates move the parameters toward a low point of the loss landscape. The practical difficulty lies in choosing step size, batch size and optimizer so that training is fast, stable and generalizes well.
How gradient descent updates parameters
Each iteration performs the update θ ← θ − η · ∇L(θ), where θ is the vector of parameters, L is the loss function and η is the learning rate. In neural networks, the gradient ∇L is computed efficiently by backpropagation, which applies the chain rule layer by layer.
One training step therefore has four parts: a forward pass that computes predictions and loss, a backward pass that computes gradients, an optimizer step that updates the weights, and resetting the accumulated gradients. One full pass over the training data is an epoch; large models are often trained for a single epoch or less over very large datasets.
Gradient descent only uses local slope information. It does not know where the global minimum is, and in deep networks the loss surface is non-convex, with saddle points, flat regions and many minima of similar quality. Fortunately, for large networks most reachable minima generalize comparably well.
Batch, stochastic and mini-batch variants
The variants differ in how much data is used to estimate each gradient. That choice trades gradient accuracy against cost per step and hardware efficiency.
| Variant | Data per update | Strength | Weakness |
|---|---|---|---|
| Batch gradient descent | Entire training set | Exact, smooth gradient | Too slow and memory-heavy for large data |
| Stochastic gradient descent | One example | Cheap steps, noise helps escape poor regions | Very noisy, poor hardware use |
| Mini-batch gradient descent | Typically 32 to several thousand examples | Good balance, efficient on GPUs | Batch size becomes a hyperparameter |
In everyday usage, SGD usually means mini-batch gradient descent. Large language model pretraining uses effective batches of millions of tokens, often assembled through gradient accumulation across several smaller batches when memory is limited.
Learning rate and other key hyperparameters
The learning rate is the single most important hyperparameter in gradient descent. Too high and the loss oscillates or diverges; too low and training crawls or stalls in flat regions.
| Hyperparameter | Typical range | Effect of increasing it |
|---|---|---|
| Learning rate, SGD | About 0.01 to 0.1 | Faster progress, risk of divergence |
| Learning rate, Adam or AdamW | About 1e-5 to 1e-3 | Same, with adaptive scaling per parameter |
| Momentum | 0.9 is common | Smoother, faster movement along consistent directions |
| Batch size | 32 to millions of tokens | Less gradient noise, more memory |
| Weight decay | About 0.01 to 0.1 with AdamW | Stronger regularization |
| Warmup steps | Hundreds to a few thousand | More stable early training |
Modern training rarely uses a constant learning rate. A schedule with a short linear warmup followed by cosine or linear decay is standard for Transformers. Fine-tuning and LoRA usually use smaller learning rates than pretraining, because the goal is to adjust an already good model rather than to learn from scratch.
Optimizers built on gradient descent
Optimizers modify the basic update to move faster through ravines and handle differently scaled parameters. All of them still rely on the gradient; they differ in how they transform it into a step.
| Optimizer | Core idea | Typical use |
|---|---|---|
| SGD with momentum | Accumulates a running average of gradients | Computer vision, well-tuned pipelines |
| Nesterov momentum | Evaluates gradient at a look-ahead position | Variant of momentum with better stability |
| RMSProp | Divides by a running average of squared gradients | Recurrent networks, older setups |
| Adam | Combines momentum and per-parameter scaling | Default for many deep learning tasks |
| AdamW | Adam with decoupled weight decay | Standard for Transformers and LLMs |
Adaptive optimizers such as Adam converge quickly with little tuning but need extra memory for two state values per parameter, which matters for very large models. Memory-saving techniques, including 8-bit optimizer states used in QLoRA, reduce this cost.
Common problems and their remedies
Most training failures appear first as abnormal loss curves, and reading those curves is a core practical skill. The table links typical symptoms to causes and fixes.
| Symptom | Likely cause | Typical fix |
|---|---|---|
| Loss becomes NaN or explodes | Learning rate too high or exploding gradients | Lower learning rate, gradient clipping |
| Loss decreases very slowly | Learning rate too low or poor initialization | Learning rate search, better initialization |
| Loss plateaus early | Saddle point or vanishing gradients | Momentum, normalization layers, residual connections |
| Training loss falls, validation loss rises | Overfitting | Regularization, early stopping, more data |
| Loss spikes in long runs | Instability with large batches or bad data | Warmup, lower peak rate, data cleaning |
A practical rule is to run a short learning rate range test before full training and to keep the validation curve under observation. Most problems that look like model defects are in fact optimization settings.
In plain words
Imagine standing on a foggy hillside and trying to reach the valley floor. You cannot see the valley, but you can feel which way the ground slopes under your feet. So you take a step downhill, feel the slope again and repeat.
Gradient descent does exactly this with a model’s errors. It checks which small change to the settings would reduce the error most, makes that change, and repeats it many times. The size of each step matters: steps that are too big overshoot, steps that are too small take forever.
A real-world example
A hospital research team fine-tunes an image classifier to detect pneumonia on chest X-rays using about 20,000 labeled images. Their first run with AdamW at a learning rate of 1e-3 shows the loss jumping erratically and the validation accuracy barely improving.
After a learning rate range test, they lower the peak rate to 1e-4, add 500 warmup steps and a cosine decay schedule, use mini-batches of 64 images and enable early stopping on validation loss. The loss now declines smoothly, and the model reaches its best validation performance after about eight epochs. The architecture never changed; only the gradient descent settings did, which is typical of how much optimization choices determine training success.