Dynamic quantization is a model compression technique in which the weights are converted to low-precision integers ahead of time, while the activations are quantized on the fly during inference using scale factors computed from the values actually observed at runtime. In Edge AI and in artificial intelligence deployment more broadly, it is the simplest form of post-training quantization because it needs no calibration dataset and no retraining.
The method is most valuable for models dominated by matrix multiplications in linear and recurrent layers, such as LSTM networks and transformer encoders running on CPUs. It typically cuts the stored size of those layers by about four times relative to 32-bit floating point and often speeds up CPU inference, at the price of a small runtime overhead for computing activation ranges.
How dynamic quantization works at inference time
Dynamic quantization splits the work between conversion time and run time. At conversion, each eligible weight tensor is mapped from FP32 to INT8 with a fixed scale and zero point, usually derived from the minimum and maximum of the weights themselves. At run time, every incoming activation tensor is scanned, its range is measured, and a fresh scale is computed for that specific input before the integer matrix multiplication runs.
The integer product is accumulated in 32-bit integer accumulators to avoid overflow, then rescaled back to floating point by multiplying with the product of the weight scale and the activation scale. Because the output returns to float, the next layer receives ordinary floating-point data, which is why the technique is easy to apply layer by layer without touching the rest of the graph.
The following table traces one quantized linear layer through its stages and shows which values are fixed and which are recomputed.
| Stage | Data type | When it is computed |
|---|---|---|
| Weight quantization | INT8 weights, FP32 scale | Once, at model conversion |
| Activation range measurement | FP32 min and max | Every forward pass |
| Activation quantization | UINT8 or INT8 activations | Every forward pass |
| Integer matrix multiplication | INT32 accumulator | Every forward pass |
| Dequantization of output | FP32 output | Every forward pass |
Key parameters and granularity choices
The behavior of dynamic quantization depends on a handful of settings that control precision, granularity and which operators are converted. The granularity of the scale matters most: a single scale per tensor is cheap but sensitive to outliers, while per-channel weight scales and per-token activation scales preserve accuracy far better in large language models.
The next table summarizes the main parameters, their common values and the effect each one has on accuracy and speed.
| Parameter | Common values | Effect |
|---|---|---|
| Weight bit width | 8 bits, sometimes 4 bits | Lower bits shrink storage but raise error |
| Activation bit width | 8 bits | Sets integer kernel type and range resolution |
| Weight scale granularity | Per tensor or per output channel | Per channel reduces error from uneven rows |
| Activation scale granularity | Per tensor, per row or per token | Finer scales handle outlier tokens |
| Symmetric or asymmetric mapping | Zero point fixed at 0 or computed | Asymmetric fits skewed ranges such as ReLU outputs |
| Eligible operator set | Linear, LSTM, GRU, embedding | Unlisted operators stay in floating point |
In practice, frameworks expose these choices through short configuration calls. PyTorch offers quantize_dynamic for linear and recurrent modules, ONNX Runtime provides a dynamic quantization tool that inserts DynamicQuantizeLinear and MatMulInteger operators, and TensorFlow Lite calls the closely related approach dynamic range quantization.
Dynamic quantization compared with other quantization methods
Dynamic quantization sits between doing nothing and fully static integer inference. Its closest neighbors are static post-training quantization, which fixes activation scales in advance using a calibration set, and quantization-aware training, which simulates quantization during training so the network learns to tolerate it. Weight-only quantization is another neighbor that never quantizes activations at all.
| Approach | Activation scales | Data needed | Typical best target |
|---|---|---|---|
| Dynamic quantization | Computed per input at runtime | None | CPU inference of linear-heavy models |
| Static post-training quantization | Fixed from calibration | Small representative set | NPUs, DSPs and microcontrollers |
| Quantization-aware training | Learned during training | Full training pipeline | Accuracy-critical compact models |
| Weight-only quantization | Not quantized | None or small set | Memory-bound LLM decoding |
The decisive difference is where the activation range comes from. Dynamic quantization adapts to each input, which protects accuracy when activation distributions vary widely, but it also prevents the fully integer pipelines that many dedicated accelerators require, because those chips expect every scale to be known at compile time.
When to choose dynamic quantization on edge devices
Dynamic quantization is the right first step when a model runs on a general-purpose CPU and its cost is dominated by large weight matrices. Speech recognition models built on LSTM layers, BERT-style text classifiers and small transformer encoders on phones or laptops are typical candidates. Reported CPU speedups commonly fall in the range of 1.5 to 3 times, depending on batch size, sequence length and kernel support.
The table below maps common deployment situations to a recommended choice.
| Situation | Recommended choice | Reason |
|---|---|---|
| Transformer encoder on ARM or x86 CPU | Dynamic quantization | Fast win without calibration data |
| CNN on mobile NPU or Edge TPU | Static full-integer quantization | Accelerator needs fixed scales |
| Microcontroller with no FPU | Static INT8 quantization | Float rescaling is too expensive |
| Large language model decoding | Weight-only or per-token dynamic | Memory bandwidth limits speed |
| Accuracy drop above tolerance | Quantization-aware training | Training recovers lost precision |
Convolutional vision models usually gain less, because their activations are large relative to their weights and the per-input range computation adds overhead that static quantization avoids. For those networks, static quantization with a short calibration run is usually the better default.
Limitations, risks and common mistakes
The main weakness of dynamic quantization is sensitivity to activation outliers. When a single extreme value stretches the measured range, every other value is squeezed into a few integer levels, which is why large language models with outlier channels often need per-token scales or techniques such as SmoothQuant. The runtime range scan also adds latency that can erase gains for small layers and tiny batch sizes.
- Expecting speedups on unsupported hardware: without integer kernels the runtime may dequantize weights and run in float, gaining only memory savings.
- Assuming the whole model is quantized: only eligible operators are converted, so attention softmax and normalization layers usually stay in FP32.
- Skipping accuracy validation: a drop that is negligible on average can concentrate in rare classes or long inputs.
- Measuring speed with a single input: latency varies with sequence length because the activation scan scales with tensor size.
Accuracy loss for 8-bit dynamic quantization of transformer encoders is typically within about one percentage point on standard benchmarks, but this is an empirical observation rather than a guarantee, and results must be confirmed on the target task.
A plain-language summary
An AI model is mostly a huge table of precise numbers. Dynamic quantization rounds those numbers to simpler whole numbers so the model takes less space and runs faster on an ordinary processor.
The fixed numbers are rounded once, in advance. The numbers that change with every request are rounded at the moment they appear, using a ruler sized to that particular request. It is a bit like a tailor who cuts the fabric beforehand but takes each customer’s measurements on the spot. The result is a smaller, quicker model with almost the same answers, and nobody has to retrain it.
An example from practice
A team building an offline note-taking app wants a text classifier based on a small BERT model to tag notes on mid-range Android phones. The FP32 model is about 440 MB and needs around 180 ms per note on the phone CPU, which feels sluggish and inflates the download.
The engineers apply dynamic quantization to every linear layer, leaving embeddings and normalization in floating point. The linear weights shrink to roughly a quarter of their size, the package falls to about 180 MB, and median latency drops to roughly 90 ms per note. On a held-out set of 5,000 labeled notes, macro F1 falls from 0.912 to 0.905. The team judges a 0.7-point loss acceptable for twice the speed and ships the quantized model without collecting any calibration data.