Dynamic quantization

On this page 7

Dynamic quantization is a model compression technique in which the weights are converted to low-precision integers ahead of time, while the activations are quantized on the fly during inference using scale factors computed from the values actually observed at runtime. In Edge AI and in artificial intelligence deployment more broadly, it is the simplest form of post-training quantization because it needs no calibration dataset and no retraining.

The method is most valuable for models dominated by matrix multiplications in linear and recurrent layers, such as LSTM networks and transformer encoders running on CPUs. It typically cuts the stored size of those layers by about four times relative to 32-bit floating point and often speeds up CPU inference, at the price of a small runtime overhead for computing activation ranges.

How dynamic quantization works at inference time

Dynamic quantization splits the work between conversion time and run time. At conversion, each eligible weight tensor is mapped from FP32 to INT8 with a fixed scale and zero point, usually derived from the minimum and maximum of the weights themselves. At run time, every incoming activation tensor is scanned, its range is measured, and a fresh scale is computed for that specific input before the integer matrix multiplication runs.

The integer product is accumulated in 32-bit integer accumulators to avoid overflow, then rescaled back to floating point by multiplying with the product of the weight scale and the activation scale. Because the output returns to float, the next layer receives ordinary floating-point data, which is why the technique is easy to apply layer by layer without touching the rest of the graph.

The following table traces one quantized linear layer through its stages and shows which values are fixed and which are recomputed.

StageData typeWhen it is computed
Weight quantizationINT8 weights, FP32 scaleOnce, at model conversion
Activation range measurementFP32 min and maxEvery forward pass
Activation quantizationUINT8 or INT8 activationsEvery forward pass
Integer matrix multiplicationINT32 accumulatorEvery forward pass
Dequantization of outputFP32 outputEvery forward pass
Stages of a dynamically quantized linear layer

Key parameters and granularity choices

The behavior of dynamic quantization depends on a handful of settings that control precision, granularity and which operators are converted. The granularity of the scale matters most: a single scale per tensor is cheap but sensitive to outliers, while per-channel weight scales and per-token activation scales preserve accuracy far better in large language models.

The next table summarizes the main parameters, their common values and the effect each one has on accuracy and speed.

ParameterCommon valuesEffect
Weight bit width8 bits, sometimes 4 bitsLower bits shrink storage but raise error
Activation bit width8 bitsSets integer kernel type and range resolution
Weight scale granularityPer tensor or per output channelPer channel reduces error from uneven rows
Activation scale granularityPer tensor, per row or per tokenFiner scales handle outlier tokens
Symmetric or asymmetric mappingZero point fixed at 0 or computedAsymmetric fits skewed ranges such as ReLU outputs
Eligible operator setLinear, LSTM, GRU, embeddingUnlisted operators stay in floating point
Main configuration parameters

In practice, frameworks expose these choices through short configuration calls. PyTorch offers quantize_dynamic for linear and recurrent modules, ONNX Runtime provides a dynamic quantization tool that inserts DynamicQuantizeLinear and MatMulInteger operators, and TensorFlow Lite calls the closely related approach dynamic range quantization.

Dynamic quantization compared with other quantization methods

Dynamic quantization sits between doing nothing and fully static integer inference. Its closest neighbors are static post-training quantization, which fixes activation scales in advance using a calibration set, and quantization-aware training, which simulates quantization during training so the network learns to tolerate it. Weight-only quantization is another neighbor that never quantizes activations at all.

ApproachActivation scalesData neededTypical best target
Dynamic quantizationComputed per input at runtimeNoneCPU inference of linear-heavy models
Static post-training quantizationFixed from calibrationSmall representative setNPUs, DSPs and microcontrollers
Quantization-aware trainingLearned during trainingFull training pipelineAccuracy-critical compact models
Weight-only quantizationNot quantizedNone or small setMemory-bound LLM decoding
Comparison of common quantization approaches

The decisive difference is where the activation range comes from. Dynamic quantization adapts to each input, which protects accuracy when activation distributions vary widely, but it also prevents the fully integer pipelines that many dedicated accelerators require, because those chips expect every scale to be known at compile time.

When to choose dynamic quantization on edge devices

Dynamic quantization is the right first step when a model runs on a general-purpose CPU and its cost is dominated by large weight matrices. Speech recognition models built on LSTM layers, BERT-style text classifiers and small transformer encoders on phones or laptops are typical candidates. Reported CPU speedups commonly fall in the range of 1.5 to 3 times, depending on batch size, sequence length and kernel support.

The table below maps common deployment situations to a recommended choice.

SituationRecommended choiceReason
Transformer encoder on ARM or x86 CPUDynamic quantizationFast win without calibration data
CNN on mobile NPU or Edge TPUStatic full-integer quantizationAccelerator needs fixed scales
Microcontroller with no FPUStatic INT8 quantizationFloat rescaling is too expensive
Large language model decodingWeight-only or per-token dynamicMemory bandwidth limits speed
Accuracy drop above toleranceQuantization-aware trainingTraining recovers lost precision
Deployment decision guide

Convolutional vision models usually gain less, because their activations are large relative to their weights and the per-input range computation adds overhead that static quantization avoids. For those networks, static quantization with a short calibration run is usually the better default.

Limitations, risks and common mistakes

The main weakness of dynamic quantization is sensitivity to activation outliers. When a single extreme value stretches the measured range, every other value is squeezed into a few integer levels, which is why large language models with outlier channels often need per-token scales or techniques such as SmoothQuant. The runtime range scan also adds latency that can erase gains for small layers and tiny batch sizes.

  • Expecting speedups on unsupported hardware: without integer kernels the runtime may dequantize weights and run in float, gaining only memory savings.
  • Assuming the whole model is quantized: only eligible operators are converted, so attention softmax and normalization layers usually stay in FP32.
  • Skipping accuracy validation: a drop that is negligible on average can concentrate in rare classes or long inputs.
  • Measuring speed with a single input: latency varies with sequence length because the activation scan scales with tensor size.

Accuracy loss for 8-bit dynamic quantization of transformer encoders is typically within about one percentage point on standard benchmarks, but this is an empirical observation rather than a guarantee, and results must be confirmed on the target task.

A plain-language summary

An AI model is mostly a huge table of precise numbers. Dynamic quantization rounds those numbers to simpler whole numbers so the model takes less space and runs faster on an ordinary processor.

The fixed numbers are rounded once, in advance. The numbers that change with every request are rounded at the moment they appear, using a ruler sized to that particular request. It is a bit like a tailor who cuts the fabric beforehand but takes each customer’s measurements on the spot. The result is a smaller, quicker model with almost the same answers, and nobody has to retrain it.

An example from practice

A team building an offline note-taking app wants a text classifier based on a small BERT model to tag notes on mid-range Android phones. The FP32 model is about 440 MB and needs around 180 ms per note on the phone CPU, which feels sluggish and inflates the download.

The engineers apply dynamic quantization to every linear layer, leaving embeddings and normalization in floating point. The linear weights shrink to roughly a quarter of their size, the package falls to about 180 MB, and median latency drops to roughly 90 ms per note. On a held-out set of 5,000 labeled notes, macro F1 falls from 0.912 to 0.905. The team judges a 0.7-point loss acceptable for twice the speed and ships the quantized model without collecting any calibration data.