Model calibration is the degree to which a model’s predicted probabilities match observed frequencies: among all predictions made with 80 percent confidence, a well-calibrated model is right about 80 percent of the time. Model evaluation in artificial intelligence treats calibration as a property separate from accuracy, because a model can rank examples well and still report confidence that cannot be trusted.
The same word also names the post-hoc procedures that repair a miscalibrated model, such as temperature scaling or isotonic regression. Calibration matters wherever probabilities drive decisions, including medical triage, fraud scoring, risk pricing, and the decision of a language model system to abstain or escalate instead of answering.
What calibration measures
Calibration measures the agreement between stated confidence and empirical accuracy across the whole range of predictions. Formally, a classifier is perfectly calibrated when, for every confidence value p, the probability that the prediction is correct given confidence p equals p.
Calibration is independent of discrimination. Discrimination, captured by metrics such as AUC or the ROC curve, asks whether positive examples receive higher scores than negative ones. A model can have an excellent AUC and poor calibration, because any monotonic distortion of the scores leaves the ranking intact while making the probabilities wrong.
Miscalibration has two directions. An overconfident model reports probabilities higher than its actual hit rate, which is typical of large modern neural networks trained to minimize cross-entropy for many epochs. An underconfident model reports probabilities that are too low, a pattern often seen after heavy regularization, label smoothing or ensembling.
How calibration is measured
Calibration is measured with reliability diagrams and summary metrics computed on held-out data that the model did not train on. The reliability diagram groups predictions into confidence bins and plots average confidence against observed accuracy; a calibrated model lies on the diagonal.
| Metric | What it computes | Strength | Weakness |
|---|---|---|---|
| Expected calibration error | Weighted average gap per confidence bin | Intuitive, widely reported | Sensitive to bin count and scheme |
| Maximum calibration error | Largest gap across bins | Shows worst-case regions | Unstable with sparse bins |
| Brier score | Mean squared error of probabilities | Proper scoring rule, no bins | Mixes calibration and discrimination |
| Negative log-likelihood | Log loss on true labels | Proper scoring rule | Dominated by confident errors |
| Adaptive calibration error | Gap with equal-count bins | More stable than fixed bins | Still depends on bin count |
Expected calibration error is usually computed with 10 to 15 equal-width bins, and values below about 0.02 to 0.05 are often considered good, although acceptable levels depend on the application. Because binning choices change the number, results are comparable only when the same binning scheme and evaluation dataset are used.
Methods for calibrating a model
Calibration methods adjust predicted probabilities after training, using a held-out calibration set that is separate from both training and test data. They leave the underlying model weights untouched and learn a small mapping from raw scores to corrected probabilities.
| Method | How it works | Data needed | Best suited to |
|---|---|---|---|
| Temperature scaling | Divides logits by one learned scalar | Small, a few thousand examples | Neural network classifiers |
| Platt scaling | Fits a logistic regression on scores | Small to moderate | Binary classifiers, support vector machines |
| Isotonic regression | Fits a non-decreasing step function | Large, often 10,000 or more | Complex distortions with enough data |
| Histogram binning | Replaces scores with bin accuracy | Moderate | Simple, interpretable correction |
| Vector or matrix scaling | Learns per-class logit transforms | Large | Many-class problems with class-specific bias |
Temperature scaling is the default for deep classifiers because it has one parameter, cannot change the predicted class, and therefore preserves accuracy exactly. A temperature above 1 softens overconfident outputs; a temperature below 1 sharpens underconfident ones. Training-time techniques such as label smoothing, focal loss and ensembling also change calibration, but their effect must still be verified on held-out data.
Calibration compared with related evaluation concepts
Calibration answers a different question from most familiar evaluation metrics, and reporting it alongside accuracy and ranking metrics gives a fuller picture of model quality. The table separates the questions each concept answers.
| Concept | Question it answers | Affected by calibration |
|---|---|---|
| Accuracy | How often is the top prediction correct | No, if the ranking is unchanged |
| AUC and ROC curve | Are positives ranked above negatives | No |
| Precision and recall | How good are decisions at a fixed threshold | Yes, through the chosen threshold |
| Calibration | Do probabilities mean what they say | Defines it |
| Overfitting | Does performance drop on unseen data | Often causes overconfidence |
Distribution shift is the most serious threat to calibration. A model calibrated on one population can become badly overconfident on another, so calibration checked on the original validation set says little about behavior after deployment unless the data remain similar.
Calibration in large language models
In large language models, calibration concerns whether token probabilities or stated confidence reflect the actual chance that an answer is correct. Pretrained base models often show reasonable calibration on multiple-choice benchmarks, while post-training with RLHF tends to make outputs more overconfident.
Practitioners therefore distinguish logit-based confidence, read from token probabilities, from verbalized confidence, where the model states a percentage in text. Verbalized confidence is easy to obtain through an API but frequently clusters at high values. Calibrated confidence underpins selective prediction, in which a system answers only above a threshold and routes the rest to a human, which can lower the hallucination rate on accepted answers.
Common mistakes when assessing calibration
Most calibration errors in practice come from evaluation design rather than from the model. The following mistakes recur across projects:
- Fitting the calibrator on test data, which reports an optimistic error that will not hold in production.
- Reporting ECE without bin details, which makes numbers from different studies incomparable.
- Checking only top-label calibration when downstream decisions use the full probability vector.
- Ignoring class imbalance, where rare classes hide large errors inside a small overall ECE.
- Assuming calibration transfers to a new population without rechecking after distribution shift.
The simple explanation
Think of a weather forecaster who says there is a 70 percent chance of rain. If it rains on about seven out of every ten such days, the forecaster is well calibrated, and people can plan around those numbers.
A machine learning model works the same way. When it says it is 90 percent sure, it should be right about nine times out of ten. A model can be good at picking answers and still boast or hedge too much. Calibration checks and fixes that, so the confidence numbers can be trusted when real decisions depend on them.
How it is used in practice
A hospital analytics team deploys a neural network that flags chest X-rays for possible pneumonia. On a held-out set of 6,000 images, the model reaches an AUC of 0.91, but its reliability diagram shows that cases scored at 0.9 are positive only about 70 percent of the time, an expected calibration error of 0.11.
The team fits temperature scaling on a separate set of 2,000 images and obtains a temperature of 1.8. Accuracy and AUC stay identical, while the calibration error drops to about 0.03. Radiologists can now use the probability directly: images above 0.85 go to an urgent review queue, and the team rechecks calibration quarterly because new scanner models can shift the image distribution.