Bayesian optimization is a sequential, model-based strategy for optimizing expensive black-box functions, in which a probabilistic surrogate model predicts the objective and an acquisition function decides where to evaluate next. In machine learning, it is the standard tool for hyperparameter tuning when every trial means training a model for minutes, hours or days, so the number of evaluations must stay small.
The method treats the objective as unknown and noisy: it has no gradient, no closed form and a high cost per query. Instead of sampling a grid or random points, Bayesian optimization learns from every result and concentrates the budget on promising regions while still checking uncertain ones. It typically reaches good configurations in tens to a few hundred evaluations, where grid search would need thousands.
How Bayesian optimization builds and uses a surrogate
Bayesian optimization works as a loop of fit, propose and evaluate. After a few initial points, often drawn by random or Latin hypercube sampling, the algorithm fits a surrogate to all observations so far, maximizes an acquisition function over the search space and evaluates the true objective at the chosen point.
The most common surrogate is a Gaussian process, which returns for every candidate point both a predicted mean and a predictive variance. The mean expresses where the model expects good results, and the variance expresses how little it knows there. The kernel, frequently a Matern 5/2 kernel with automatic relevance determination, encodes assumptions about smoothness and about which dimensions matter.
The steps below summarize one iteration and the role of each component.
| Step | What happens | Typical cost |
|---|---|---|
| Initial design | Evaluate 5 to 20 space-filling points | Several full objective evaluations |
| Fit surrogate | Condition the model on all observations | Cubic in the number of points for a GP |
| Maximize acquisition | Search the cheap acquisition surface | Seconds to minutes |
| Evaluate objective | Train and validate the real model | Minutes to days |
| Update history | Append the new result and repeat | Negligible |
The economics are the point: the method spends extra computation on the surrogate to save expensive evaluations of the objective. When one evaluation takes a millisecond, this overhead is not worth paying.
Acquisition functions and the exploration balance
An acquisition function converts the surrogate’s mean and variance into a single score that ranks candidate points. Its central job is the exploration-exploitation balance: sampling near the current best result versus sampling where uncertainty is large and a better optimum might hide.
The following table compares the acquisition functions that practitioners meet most often.
| Acquisition function | Core idea | Practical behavior |
|---|---|---|
| Expected improvement | Average gain over the best value so far | Robust default, mildly greedy |
| Probability of improvement | Chance of beating the incumbent | Tends to over-exploit without a margin |
| Upper confidence bound | Mean plus a multiple of standard deviation | Explicit exploration weight, theoretical guarantees |
| Thompson sampling | Optimize one random draw from the posterior | Natural fit for parallel batches |
| Knowledge gradient | Value of information for the final choice | Strong with noise, costlier to compute |
Expected improvement remains the default in most libraries because it needs no tuning and behaves well in noisy settings. Upper confidence bound exposes a parameter, often called beta or kappa, that directly sets how adventurous the search is.
How it compares with other tuning methods
Bayesian optimization differs from grid search and random search because it uses past results to choose the next trial. Grid and random search are embarrassingly parallel and simple, but they waste evaluations in regions that earlier trials already showed to be poor.
The comparison below places the method among the main alternatives used for hyperparameter tuning and AutoML.
| Method | Uses past results | Parallelism | Best suited for |
|---|---|---|---|
| Grid search | No | Trivial | One or two parameters, cheap models |
| Random search | No | Trivial | Moderate budgets, many dimensions |
| Bayesian optimization | Yes, via a surrogate | Limited, needs batch methods | Expensive trials, under about 20 dimensions |
| Hyperband and ASHA | Only through early stopping | High | Models with cheap partial training |
| Evolutionary search | Yes, via a population | High | Large, irregular spaces |
In practice, these approaches are often combined. BOHB pairs a Bayesian surrogate with Hyperband’s early stopping, and the tree-structured Parzen estimator used by several popular tuning libraries replaces the Gaussian process with density models that handle conditional and categorical parameters more easily.
Configuring a Bayesian optimization run
A successful run depends more on defining the search space well than on choosing an exotic acquisition function. Ranges that span orders of magnitude, such as learning rate or regularization strength, should be searched on a logarithmic scale, and irrelevant parameters should be fixed rather than tuned.
The decision table below links typical situations to sensible configuration choices.
| Situation | Recommended choice | Reason |
|---|---|---|
| Fewer than 10 continuous parameters | GP surrogate with expected improvement | Best sample efficiency |
| Many categorical or conditional parameters | Tree-structured Parzen estimator or random forest surrogate | Handles mixed spaces natively |
| Noisy validation scores | Noise-aware GP, repeated folds | Prevents chasing lucky results |
| Several GPUs available | Batch acquisition or Thompson sampling | Proposes diverse points at once |
| Training can be stopped early | Multi-fidelity method such as BOHB | Discards poor trials cheaply |
Budgets of 10 to 20 times the number of dimensions are a reasonable starting point. The objective should be a cross-validated or held-out metric, never the training loss, otherwise the search optimizes overfitting.
Limitations and common mistakes
Bayesian optimization degrades in high-dimensional spaces, generally beyond about 20 effective dimensions, because the surrogate cannot learn a reliable model from few points. Standard Gaussian processes also scale cubically with the number of observations, which makes runs with thousands of evaluations slow unless sparse or ensemble surrogates are used.
- Overfitting the validation set: hundreds of trials against one split can select a configuration that does not generalize.
- Poor kernel or scale assumptions: a linear scale for learning rate hides the region where good values live.
- Ignoring noise: treating a single noisy score as exact makes the search exploit random luck.
- Sequential bottleneck: the classic loop proposes one point at a time and idles parallel hardware.
- Overhead on cheap problems: when trials take seconds, random search often wins on wall-clock time.
A common misconception is that the method guarantees the global optimum. It provides an efficient, well-informed search, and its results remain only as good as the surrogate’s assumptions and the evaluation budget.
In simple terms
Imagine searching for the best spot to drill for water when each test well costs a fortune. You would not drill everywhere. You would drill a few holes, sketch a map of where water probably is and how sure you are, then choose the next hole where the map looks most promising or most uncertain.
Bayesian optimization does exactly this for computer settings. Each test is expensive, so it keeps a running guess about which settings work well and picks the next test carefully. The result is a good answer with far fewer costly tries.
A practical example
A data science team at an insurance company trains a gradient boosting model to predict claim fraud. Each training run on 40 million records with five-fold cross-validation takes about 45 minutes, and the model has seven important hyperparameters, including learning rate, tree depth, subsampling ratio and two regularization terms.
A full grid with five values per parameter would require 78,125 runs, far beyond any budget. The team instead defines log-scaled ranges, starts with 14 Latin hypercube points and runs Bayesian optimization with a Gaussian process surrogate and expected improvement for a total of 80 trials over three days.
The best configuration raises the validation area under the precision-recall curve from 0.61 with default settings to 0.68. A final check on an untouched test set confirms the gain, and the team records the search history so that the next retraining starts from the already explored promising region.