Post-training quantization (PTQ) is the process of converting an already trained neural network from 32-bit or 16-bit floating point to a lower-precision numeric format, typically 8-bit or 4-bit integers, without retraining the model. In Edge AI, it is the most common first step for making a model small and fast enough to run on phones, PCs, microcontrollers and embedded accelerators.
PTQ trades a small, measurable loss of accuracy for roughly 2x to 8x smaller weights, lower memory bandwidth and access to integer arithmetic units such as NPUs and DSPs. Because it needs only the trained model and, in most variants, a small calibration dataset, it is far cheaper than quantization-aware training.
How post-training quantization maps floats to integers
PTQ works by choosing, for each tensor or channel, a scale factor and a zero point that map a floating-point range onto a small integer grid. A real value x becomes q = round(x / scale) + zero_point, clipped to the integer range, and is recovered approximately as x ≈ scale × (q − zero_point). The difference between x and its reconstruction is the quantization error.
The central decision is the clipping range. A wide range covers rare outliers but spends the grid coarsely on common values; a narrow range resolves typical values finely but saturates outliers. Choosing this range is called quantization calibration, and it is where most PTQ accuracy is won or lost.
The process usually follows a fixed sequence. The table below shows the typical stages of a PTQ pipeline and what each one produces.
| Stage | What happens | Output |
|---|---|---|
| Graph preparation | Batch normalization is folded into convolutions; operators are fused | Simplified float graph |
| Weight quantization | Scales are computed from the known weight values | Integer weights with scales |
| Calibration | Representative inputs run through the model to record activation ranges | Activation scales and zero points |
| Conversion | Float operators are replaced by integer kernels | Deployable quantized model |
| Validation | Accuracy is compared with the float baseline | Accuracy delta per task |
Variants of PTQ and their parameters
PTQ is a family of methods rather than one algorithm. Variants differ in what is quantized (weights only or weights and activations), when activation scales are computed (ahead of time or at run time) and how finely scales are assigned (per tensor, per channel or per group).
The following table compares the most widely used variants and the conditions under which each fits best.
| Variant | Quantized tensors | Calibration data | Typical use |
|---|---|---|---|
| Static INT8 | Weights and activations | Required, often 100-1,000 samples | CNNs on NPUs and DSPs |
| Dynamic quantization | Weights; activations at run time | Not required | RNNs and transformers on CPUs |
| Weight-only INT4 or INT8 | Weights only | Small or none | Local LLMs limited by memory bandwidth |
| Mixed precision | Sensitive layers kept at higher precision | Required | Models with fragile layers |
| FP8 or FP16 casting | Weights and activations | Optional | GPUs with native low-precision floats |
Granularity matters as much as bit width. Per-channel quantization of convolution weights is now the default because channel ranges can differ by orders of magnitude. For large language models, group-wise scales of 32 to 128 weights make 4-bit formats practical, and methods such as GPTQ, AWQ and SmoothQuant add error-compensating steps on top of basic PTQ.
PTQ compared with quantization-aware training
The main alternative to PTQ is quantization-aware training (QAT), which simulates quantization during training so the network learns weights that tolerate rounding. QAT usually recovers more accuracy at low bit widths, but it requires the training pipeline, labeled data and significant compute, which many deployment teams do not have.
| Criterion | Post-training quantization | Quantization-aware training |
|---|---|---|
| Training required | No | Yes, fine-tuning with fake quantization |
| Data needed | Small unlabeled calibration set | Full or partial labeled training set |
| Time to deploy | Minutes to hours | Hours to days |
| Accuracy at INT8 | Usually within about 1% of float | Usually near float accuracy |
| Accuracy at 4 bits or below | Often degrades without advanced methods | Generally more robust |
In practice, teams start with PTQ and move to QAT only when validation shows an unacceptable gap. PTQ also combines well with other compression methods such as pruning, weight clustering and operator fusion, which the model conversion toolchain often applies in the same pass.
Applying PTQ on edge hardware
Successful PTQ depends on matching the quantization scheme to the target hardware delegate. Many NPUs accept only symmetric per-tensor INT8 activations; if the converted model uses an unsupported scheme, the runtime silently falls back to the CPU and the expected speedup disappears.
The decision table below summarizes common deployment situations and a reasonable PTQ strategy for each.
| Situation | Recommended approach | Reason |
|---|---|---|
| Microcontroller with no FPU | Full integer static INT8 | Integer-only kernels are mandatory |
| Smartphone NPU | Static INT8, per-channel weights | Matches common delegate requirements |
| Laptop CPU running a local LLM | Weight-only 4-bit with group scales | Inference is memory-bound |
| Accuracy-critical vision model | Mixed precision with sensitive layers in FP16 | Limits error in fragile layers |
| No representative data available | Dynamic or weight-only quantization | Avoids activation calibration |
The calibration set must resemble production inputs. Images from a different camera, lighting or resolution produce misleading activation ranges, and a few hundred well-chosen samples usually outperform thousands of unrepresentative ones.
Limitations and common failure modes
PTQ is not free of risk. The errors below account for most failed deployments, and nearly all of them are detectable with layer-wise comparison against the float model.
- Activation outliers: a few extreme values, common in transformer layers, stretch the range and destroy resolution for everything else.
- Depthwise convolutions: compact architectures such as MobileNet have channels with very different ranges and suffer badly under per-tensor scales.
- Unrepresentative calibration: ranges recorded on clean data clip real-world inputs and cause unpredictable errors.
- Aggregate metrics hide damage: top-1 accuracy may hold while rare classes or long-context tasks degrade sharply.
- Operator fallback: unsupported quantized operators force costly conversions between integer and float at run time.
A frequent misconception is that fewer bits always mean faster inference. Speed depends on whether the hardware has native kernels for that format; 4-bit weights on a processor without 4-bit support are unpacked to 8 or 16 bits and save memory but not compute.
A plain-language summary
A trained AI model stores millions or billions of numbers with many decimal places. Post-training quantization rounds those numbers to a much shorter form after training is finished, much like saving a photo at a lower but still good quality setting.
The model becomes several times smaller and faster, so it can run on a phone or a small device instead of a distant server. The cost is a slight loss of precision, which engineers measure carefully. The key idea is simple: keep the learned knowledge, but store it with fewer digits.
An example from practice
A team building a smart doorbell needs a person-detection model to run on a low-power vision chip with an INT8 accelerator. The float model is about 14 MB and reaches 91.2% accuracy on its validation set, but the chip has only a few megabytes of fast memory available.
The engineers apply static INT8 post-training quantization with per-channel weight scales and calibrate it on 500 frames recorded by real doorbells at day, dusk and night. The quantized model shrinks to about 3.6 MB, runs fully on the accelerator and reaches 90.6% accuracy. Inference time falls from roughly 120 ms on the CPU to under 20 ms.
An earlier attempt calibrated on studio photos had lost several points of accuracy at night. Representative calibration data closed most of that gap, and the team shipped without needing quantization-aware training.