A hyperparameter is a configuration value set before training begins that controls how a machine learning model learns, rather than what it learns. In artificial intelligence, typical examples are the learning rate, batch size, number of epochs, weight decay and the number of layers in a network. Unlike model parameters, hyperparameters are not updated by gradient descent; they are fixed by a practitioner or found through a systematic search.
Hyperparameters matter because the same architecture and the same data can produce a strong model or a useless one depending on these choices. In training, they govern convergence speed, stability and generalization, and they often account for a larger share of the compute budget than the final training run itself.
How hyperparameters differ from model parameters
The difference between a hyperparameter and a parameter is who sets the value and when. Model parameters, such as weights and biases, are learned from data by an optimizer during training. Hyperparameters sit one level above: they define the optimizer, the architecture and the training procedure that produce those parameters.
This distinction creates a nested optimization problem. The inner loop minimizes the loss function over the parameters, while the outer loop searches over hyperparameters and judges each candidate on a held-out validation set. Using the test set for this outer search leaks information and produces optimistic performance estimates.
| Property | Model parameter | Hyperparameter |
|---|---|---|
| Examples | Weights, biases, embeddings | Learning rate, batch size, depth |
| How it is set | Learned by the optimizer | Chosen by a person or a search |
| When it is set | During training | Before or between runs |
| Typical count | Millions to trillions | A few to a few dozen |
| Judged on | Training loss | Validation metric |
Main categories of hyperparameters
Hyperparameters fall into three broad families: those of the optimizer, those of the model architecture and those of the data pipeline and regularization. Each family affects training through a different mechanism, so tuning them together requires an understanding of how they interact.
| Hyperparameter | Typical range | What it controls |
|---|---|---|
| Learning rate | 1e-5 to 1e-1 | Step size of each weight update |
| Batch size | 8 to several million tokens | Gradient noise and hardware use |
| Weight decay | 0 to 0.1 | Strength of weight shrinkage |
| Warmup steps | 0.1% to 5% of training | Stability in the first updates |
| Dropout rate | 0 to 0.5 | Random deactivation of units |
| Adam beta values | 0.9 and 0.95 to 0.999 | Momentum and variance averaging |
| Number of layers | A few to over 100 | Model capacity and depth |
The ranges above are indicative, not universal. The right learning rate, for example, depends on the optimizer, model width and batch size, and fine-tuning a pretrained model usually calls for a rate one to two orders of magnitude smaller than pretraining.
How hyperparameters interact during training
Hyperparameters are rarely independent, and changing one often shifts the optimal value of another. The best-known coupling is between learning rate and batch size: larger batches produce less noisy gradients, which often tolerate a larger learning rate, up to a critical batch size beyond which extra data per step brings diminishing returns.
Other couplings are subtler. Weight decay in AdamW interacts with the learning rate because the effective shrinkage per step is their product. The learning rate schedule interacts with the total number of steps, so a schedule tuned for a short run may be wrong for a long one. Techniques such as maximal update parametrization (muP) were developed to make some optimal hyperparameters transfer from small proxy models to large ones.
- Learning rate and batch size: often scaled together, but not linearly at every size.
- Learning rate and warmup: higher peak rates usually need longer warmup to avoid loss spikes.
- Model width and learning rate: under standard parametrization, wider models need smaller rates.
- Epochs and regularization: more passes over data increase the need for dropout or weight decay.
Methods for hyperparameter tuning
Hyperparameter tuning, also called hyperparameter optimization, is the systematic search for values that maximize a validation metric within a fixed compute budget. The choice of method depends on how expensive one training run is and how many hyperparameters are in play.
| Method | How it works | Best suited for |
|---|---|---|
| Grid search | Tests every combination on a fixed grid | One or two hyperparameters, cheap runs |
| Random search | Samples values from defined distributions | Many hyperparameters, moderate budgets |
| Bayesian optimization | Models the metric and picks promising points | Expensive runs, under about 20 dimensions |
| Hyperband and ASHA | Stops weak candidates early | Large searches with early signal |
| Population-based training | Mutates settings during parallel runs | Schedules and reinforcement learning |
| Scaling-law transfer | Tunes small proxies, extrapolates | Large language model pretraining |
Random search usually beats grid search in practice because only a few hyperparameters strongly affect the result, and random sampling explores more distinct values of each. Search spaces for scale-like values, such as learning rate and weight decay, should be defined on a logarithmic scale, not a linear one.
Common mistakes and limitations
Most hyperparameter failures come from poor experimental hygiene rather than from a bad search algorithm. The table below lists frequent symptoms and their usual causes.
| Symptom | Likely cause | Typical fix |
|---|---|---|
| Loss diverges or turns NaN | Learning rate too high | Lower the rate, add warmup or clipping |
| Loss falls very slowly | Learning rate too low | Run a learning rate range test |
| Validation loss rises while training loss falls | Overfitting | Add regularization or early stopping |
| Great validation score, poor test score | Overtuning on the validation set | Use a fresh holdout or cross-validation |
| Results change between seeds | High run-to-run variance | Average over several seeds |
A further limitation is cost. For large models, a full search is often unaffordable, so teams rely on published defaults, proxy experiments and scaling rules. These shortcuts work well in familiar regimes but can mislead when the architecture, data mixture or precision format changes.
The short version
A hyperparameter is a setting chosen before the learning starts. The model then learns everything else by itself from data. Think of baking bread: the recipe adjusts itself as the dough rises, but someone must first choose the oven temperature and baking time.
If those choices are wrong, even good ingredients give a poor result. Engineers therefore try several settings, compare the outcomes on data the model has not seen, and keep the combination that works best.
Where it makes a difference
A machine learning team fine-tunes a 7-billion-parameter language model to classify customer support tickets into 40 categories. The first run uses a learning rate of 1e-4 copied from a pretraining script, and after two epochs the model has forgotten much of its general language ability while validation accuracy stalls at 78%.
The team runs a random search of 24 trials over learning rate from 1e-6 to 1e-4 on a log scale, warmup ratio and weight decay, each trial trained on 10% of the data. The best region centers on a learning rate near 2e-5 with 3% warmup. A full run with those values reaches 91% validation accuracy, and the result holds on an untouched test set, which confirms that the gain came from better hyperparameters rather than from overtuning.