Hyperparameter

On this page 7

A hyperparameter is a configuration value set before training begins that controls how a machine learning model learns, rather than what it learns. In artificial intelligence, typical examples are the learning rate, batch size, number of epochs, weight decay and the number of layers in a network. Unlike model parameters, hyperparameters are not updated by gradient descent; they are fixed by a practitioner or found through a systematic search.

Hyperparameters matter because the same architecture and the same data can produce a strong model or a useless one depending on these choices. In training, they govern convergence speed, stability and generalization, and they often account for a larger share of the compute budget than the final training run itself.

How hyperparameters differ from model parameters

The difference between a hyperparameter and a parameter is who sets the value and when. Model parameters, such as weights and biases, are learned from data by an optimizer during training. Hyperparameters sit one level above: they define the optimizer, the architecture and the training procedure that produce those parameters.

This distinction creates a nested optimization problem. The inner loop minimizes the loss function over the parameters, while the outer loop searches over hyperparameters and judges each candidate on a held-out validation set. Using the test set for this outer search leaks information and produces optimistic performance estimates.

PropertyModel parameterHyperparameter
ExamplesWeights, biases, embeddingsLearning rate, batch size, depth
How it is setLearned by the optimizerChosen by a person or a search
When it is setDuring trainingBefore or between runs
Typical countMillions to trillionsA few to a few dozen
Judged onTraining lossValidation metric
Parameters and hyperparameters compared

Main categories of hyperparameters

Hyperparameters fall into three broad families: those of the optimizer, those of the model architecture and those of the data pipeline and regularization. Each family affects training through a different mechanism, so tuning them together requires an understanding of how they interact.

HyperparameterTypical rangeWhat it controls
Learning rate1e-5 to 1e-1Step size of each weight update
Batch size8 to several million tokensGradient noise and hardware use
Weight decay0 to 0.1Strength of weight shrinkage
Warmup steps0.1% to 5% of trainingStability in the first updates
Dropout rate0 to 0.5Random deactivation of units
Adam beta values0.9 and 0.95 to 0.999Momentum and variance averaging
Number of layersA few to over 100Model capacity and depth
Common hyperparameters and their typical ranges

The ranges above are indicative, not universal. The right learning rate, for example, depends on the optimizer, model width and batch size, and fine-tuning a pretrained model usually calls for a rate one to two orders of magnitude smaller than pretraining.

How hyperparameters interact during training

Hyperparameters are rarely independent, and changing one often shifts the optimal value of another. The best-known coupling is between learning rate and batch size: larger batches produce less noisy gradients, which often tolerate a larger learning rate, up to a critical batch size beyond which extra data per step brings diminishing returns.

Other couplings are subtler. Weight decay in AdamW interacts with the learning rate because the effective shrinkage per step is their product. The learning rate schedule interacts with the total number of steps, so a schedule tuned for a short run may be wrong for a long one. Techniques such as maximal update parametrization (muP) were developed to make some optimal hyperparameters transfer from small proxy models to large ones.

  • Learning rate and batch size: often scaled together, but not linearly at every size.
  • Learning rate and warmup: higher peak rates usually need longer warmup to avoid loss spikes.
  • Model width and learning rate: under standard parametrization, wider models need smaller rates.
  • Epochs and regularization: more passes over data increase the need for dropout or weight decay.

Methods for hyperparameter tuning

Hyperparameter tuning, also called hyperparameter optimization, is the systematic search for values that maximize a validation metric within a fixed compute budget. The choice of method depends on how expensive one training run is and how many hyperparameters are in play.

MethodHow it worksBest suited for
Grid searchTests every combination on a fixed gridOne or two hyperparameters, cheap runs
Random searchSamples values from defined distributionsMany hyperparameters, moderate budgets
Bayesian optimizationModels the metric and picks promising pointsExpensive runs, under about 20 dimensions
Hyperband and ASHAStops weak candidates earlyLarge searches with early signal
Population-based trainingMutates settings during parallel runsSchedules and reinforcement learning
Scaling-law transferTunes small proxies, extrapolatesLarge language model pretraining
Tuning methods and when each fits

Random search usually beats grid search in practice because only a few hyperparameters strongly affect the result, and random sampling explores more distinct values of each. Search spaces for scale-like values, such as learning rate and weight decay, should be defined on a logarithmic scale, not a linear one.

Common mistakes and limitations

Most hyperparameter failures come from poor experimental hygiene rather than from a bad search algorithm. The table below lists frequent symptoms and their usual causes.

SymptomLikely causeTypical fix
Loss diverges or turns NaNLearning rate too highLower the rate, add warmup or clipping
Loss falls very slowlyLearning rate too lowRun a learning rate range test
Validation loss rises while training loss fallsOverfittingAdd regularization or early stopping
Great validation score, poor test scoreOvertuning on the validation setUse a fresh holdout or cross-validation
Results change between seedsHigh run-to-run varianceAverage over several seeds
Symptoms, likely causes and fixes

A further limitation is cost. For large models, a full search is often unaffordable, so teams rely on published defaults, proxy experiments and scaling rules. These shortcuts work well in familiar regimes but can mislead when the architecture, data mixture or precision format changes.

The short version

A hyperparameter is a setting chosen before the learning starts. The model then learns everything else by itself from data. Think of baking bread: the recipe adjusts itself as the dough rises, but someone must first choose the oven temperature and baking time.

If those choices are wrong, even good ingredients give a poor result. Engineers therefore try several settings, compare the outcomes on data the model has not seen, and keep the combination that works best.

Where it makes a difference

A machine learning team fine-tunes a 7-billion-parameter language model to classify customer support tickets into 40 categories. The first run uses a learning rate of 1e-4 copied from a pretraining script, and after two epochs the model has forgotten much of its general language ability while validation accuracy stalls at 78%.

The team runs a random search of 24 trials over learning rate from 1e-6 to 1e-4 on a log scale, warmup ratio and weight decay, each trial trained on 10% of the data. The best region centers on a learning rate near 2e-5 with 3% warmup. A full run with those values reaches 91% validation accuracy, and the result holds on an untouched test set, which confirms that the gain came from better hyperparameters rather than from overtuning.