Quantization calibration

Also known as: calibration dataset

On this page 7

Quantization calibration is the step in post-training quantization that runs a small representative dataset through a full-precision model, records the range of values each tensor produces, and uses those statistics to choose the scale and zero-point that map floating-point numbers to integers. In edge AI, it is what allows a model to run in INT8 or lower precision on phones and microcontrollers without retraining.

The data used for this step is called the calibration dataset. Weights are known in advance and can be quantized directly, but activations depend on the input, so their ranges must be estimated from real examples. A good calibration keeps accuracy within about one percentage point of the original model for many vision networks; a poor one can make the model unusable.

How calibration determines quantization parameters

Calibration works by observing activations, then fixing a clipping range for each quantized tensor. During the calibration pass, observer modules attached to each layer collect statistics such as minimum, maximum or a histogram of values. After the pass, each observer converts its statistics into a range from a lower bound to an upper bound.

For asymmetric 8-bit quantization, the scale is the range width divided by 255, and the zero-point is the integer that represents real zero. For symmetric quantization, the range is centered on zero and the zero-point is fixed, which simplifies integer arithmetic on many accelerators. Any value outside the chosen range is clipped to the nearest limit.

The core trade-off is between clipping error and rounding error. A wide range avoids clipping rare large values but spreads 256 levels thinly, so small values lose precision. A narrow range gives fine resolution but cuts off outliers.

TensorNeeds calibration dataParameters setTypical granularity
WeightsNoScale, sometimes zero-pointPer-channel
ActivationsYesScale and zero-pointPer-tensor
BiasesNoDerived from input and weight scalesPer-channel, 32-bit
Model inputsYes, or from preprocessingScale and zero-pointPer-tensor
What calibration fixes for each tensor type

Calibration methods and range estimators

Calibration methods differ in how they turn observed values into a range, and the best choice depends on the shape of the activation distribution. Distributions with long tails, common after attention layers or in transformers, punish naive methods.

MethodHow the range is chosenStrengthWeakness
Min-maxAbsolute observed minimum and maximumSimple, no clippingOne outlier wastes resolution
Moving-average min-maxSmoothed extremes across batchesLess sensitive to single batchesStill follows outliers
PercentileFor example the 99.9th or 99.99th percentileIgnores rare extremesPercentile must be tuned
Entropy (KL divergence)Range that best preserves the distributionStrong for convolutional networksSlower, needs histograms
MSE minimizationRange with lowest quantization errorBalances clipping and roundingSearch adds compute time
Common range estimation methods

For large language models, calibration often goes further than range estimation. Methods such as GPTQ and AWQ use calibration activations to adjust or rescale the weights themselves, and SmoothQuant uses them to move activation outliers into the weights before quantizing.

Building a good calibration dataset

A calibration dataset must be representative of real inference inputs, not merely large. Labels are not needed, because calibration only observes activations. What matters is that the samples cover the lighting, accents, vocabulary or sensor conditions the deployed device will meet.

Sample counts are modest. Image classifiers usually need a few hundred samples, and returns diminish quickly beyond about 1,000. LLM weight quantization methods commonly use 128 sequences of about 2,048 tokens. The table summarizes typical starting points, which should be confirmed on a validation set.

Model typeTypical sample countMain coverage concern
Image classifier100 to 1,000 imagesClasses, lighting, camera types
Object detector300 to 1,000 imagesObject scales and scene density
Keyword spotting modelSeveral hundred audio clipsSpeakers, noise, microphones
Large language model64 to 512 text sequencesDomain, language, sequence length
Typical calibration set sizes
  • Apply exactly the same preprocessing as in deployment, including resizing, normalization and tokenization.
  • Keep calibration samples separate from the test set, so accuracy measurements stay honest.
  • Include edge conditions the product will meet, such as low light or background noise.

Calibration compared with dynamic quantization and quantization-aware training

Calibration is the defining feature of static post-training quantization, and the alternatives either skip it or replace it with training. The choice depends on accuracy targets, hardware support and available engineering effort.

ApproachWhen ranges are setData neededEdge suitability
Static post-training quantizationOnce, during calibrationUnlabeled samplesBest for integer-only accelerators
Dynamic quantizationAt runtime, per inputNoneGood on CPUs, weak on NPUs
Quantization-aware trainingLearned during fine-tuningLabeled training dataHighest accuracy, highest cost
Weight-only quantizationNo activation rangesOptional for weight tuningCommon for local LLMs
How activation ranges are obtained

A practical sequence is to start with static calibration, measure the accuracy drop, and move to per-channel weights, a better range method or quantization-aware training only if the drop exceeds the product tolerance.

Failure modes and how to diagnose them

Most calibration failures come from a mismatch between calibration data and real inputs or from a few sensitive layers. Diagnosis usually compares per-layer outputs of the float and quantized models to find where error accumulates.

  • Preprocessing mismatch: calibrating on images scaled to 0 to 1 while the device feeds 0 to 255 produces ranges that are wrong by a large factor.
  • Outlier-dominated ranges: min-max on tensors with rare spikes leaves most values in a few integer levels; percentile or MSE methods help.
  • Too few or too narrow samples: a model calibrated on daylight photos degrades at night.
  • Sensitive layers: first and last layers, or attention softmax inputs, may need to stay in higher precision, which is the basis of mixed-precision quantization.

A non-technical view

To make an AI model small enough for a phone, its numbers are converted from precise decimals to a short list of whole numbers. Before doing that, the converter has to know how big and how small the numbers usually get. Quantization calibration is that measuring step.

It is like setting the scale on a kitchen measuring jug before pouring. If the markings cover too wide a range, small amounts are hard to read; if they cover too little, the jug overflows. The key takeaway: calibration chooses the right markings using real sample inputs, so the smaller model stays almost as accurate as the original.

A concrete example

A team builds a people-detection model for a battery-powered doorbell camera with an integer-only neural processor. The float model reaches 91 percent mean average precision. After quantizing to INT8 with min-max calibration on 200 daytime images, accuracy falls to 78 percent, and night-time detections become unreliable.

Per-layer analysis shows that a few activation outliers stretch the ranges in early layers. The team rebuilds the calibration dataset with 600 images covering day, dusk and infrared night footage, switches to a percentile method at 99.99 percent, and keeps the final detection layer in 16-bit precision. The quantized model reaches 89.8 percent mean average precision, runs about three times faster than the float version on the device, and fits within the camera memory budget.