Post-training quantization

Also known as: PTQ

On this page 7

Post-training quantization (PTQ) is the process of converting an already trained neural network from 32-bit or 16-bit floating point to a lower-precision numeric format, typically 8-bit or 4-bit integers, without retraining the model. In Edge AI, it is the most common first step for making a model small and fast enough to run on phones, PCs, microcontrollers and embedded accelerators.

PTQ trades a small, measurable loss of accuracy for roughly 2x to 8x smaller weights, lower memory bandwidth and access to integer arithmetic units such as NPUs and DSPs. Because it needs only the trained model and, in most variants, a small calibration dataset, it is far cheaper than quantization-aware training.

How post-training quantization maps floats to integers

PTQ works by choosing, for each tensor or channel, a scale factor and a zero point that map a floating-point range onto a small integer grid. A real value x becomes q = round(x / scale) + zero_point, clipped to the integer range, and is recovered approximately as x ≈ scale × (q − zero_point). The difference between x and its reconstruction is the quantization error.

The central decision is the clipping range. A wide range covers rare outliers but spends the grid coarsely on common values; a narrow range resolves typical values finely but saturates outliers. Choosing this range is called quantization calibration, and it is where most PTQ accuracy is won or lost.

The process usually follows a fixed sequence. The table below shows the typical stages of a PTQ pipeline and what each one produces.

StageWhat happensOutput
Graph preparationBatch normalization is folded into convolutions; operators are fusedSimplified float graph
Weight quantizationScales are computed from the known weight valuesInteger weights with scales
CalibrationRepresentative inputs run through the model to record activation rangesActivation scales and zero points
ConversionFloat operators are replaced by integer kernelsDeployable quantized model
ValidationAccuracy is compared with the float baselineAccuracy delta per task
Typical stages of a post-training quantization pipeline

Variants of PTQ and their parameters

PTQ is a family of methods rather than one algorithm. Variants differ in what is quantized (weights only or weights and activations), when activation scales are computed (ahead of time or at run time) and how finely scales are assigned (per tensor, per channel or per group).

The following table compares the most widely used variants and the conditions under which each fits best.

VariantQuantized tensorsCalibration dataTypical use
Static INT8Weights and activationsRequired, often 100-1,000 samplesCNNs on NPUs and DSPs
Dynamic quantizationWeights; activations at run timeNot requiredRNNs and transformers on CPUs
Weight-only INT4 or INT8Weights onlySmall or noneLocal LLMs limited by memory bandwidth
Mixed precisionSensitive layers kept at higher precisionRequiredModels with fragile layers
FP8 or FP16 castingWeights and activationsOptionalGPUs with native low-precision floats
Common post-training quantization variants

Granularity matters as much as bit width. Per-channel quantization of convolution weights is now the default because channel ranges can differ by orders of magnitude. For large language models, group-wise scales of 32 to 128 weights make 4-bit formats practical, and methods such as GPTQ, AWQ and SmoothQuant add error-compensating steps on top of basic PTQ.

PTQ compared with quantization-aware training

The main alternative to PTQ is quantization-aware training (QAT), which simulates quantization during training so the network learns weights that tolerate rounding. QAT usually recovers more accuracy at low bit widths, but it requires the training pipeline, labeled data and significant compute, which many deployment teams do not have.

CriterionPost-training quantizationQuantization-aware training
Training requiredNoYes, fine-tuning with fake quantization
Data neededSmall unlabeled calibration setFull or partial labeled training set
Time to deployMinutes to hoursHours to days
Accuracy at INT8Usually within about 1% of floatUsually near float accuracy
Accuracy at 4 bits or belowOften degrades without advanced methodsGenerally more robust
Post-training quantization versus quantization-aware training

In practice, teams start with PTQ and move to QAT only when validation shows an unacceptable gap. PTQ also combines well with other compression methods such as pruning, weight clustering and operator fusion, which the model conversion toolchain often applies in the same pass.

Applying PTQ on edge hardware

Successful PTQ depends on matching the quantization scheme to the target hardware delegate. Many NPUs accept only symmetric per-tensor INT8 activations; if the converted model uses an unsupported scheme, the runtime silently falls back to the CPU and the expected speedup disappears.

The decision table below summarizes common deployment situations and a reasonable PTQ strategy for each.

SituationRecommended approachReason
Microcontroller with no FPUFull integer static INT8Integer-only kernels are mandatory
Smartphone NPUStatic INT8, per-channel weightsMatches common delegate requirements
Laptop CPU running a local LLMWeight-only 4-bit with group scalesInference is memory-bound
Accuracy-critical vision modelMixed precision with sensitive layers in FP16Limits error in fragile layers
No representative data availableDynamic or weight-only quantizationAvoids activation calibration
Choosing a PTQ strategy by deployment target

The calibration set must resemble production inputs. Images from a different camera, lighting or resolution produce misleading activation ranges, and a few hundred well-chosen samples usually outperform thousands of unrepresentative ones.

Limitations and common failure modes

PTQ is not free of risk. The errors below account for most failed deployments, and nearly all of them are detectable with layer-wise comparison against the float model.

  • Activation outliers: a few extreme values, common in transformer layers, stretch the range and destroy resolution for everything else.
  • Depthwise convolutions: compact architectures such as MobileNet have channels with very different ranges and suffer badly under per-tensor scales.
  • Unrepresentative calibration: ranges recorded on clean data clip real-world inputs and cause unpredictable errors.
  • Aggregate metrics hide damage: top-1 accuracy may hold while rare classes or long-context tasks degrade sharply.
  • Operator fallback: unsupported quantized operators force costly conversions between integer and float at run time.

A frequent misconception is that fewer bits always mean faster inference. Speed depends on whether the hardware has native kernels for that format; 4-bit weights on a processor without 4-bit support are unpacked to 8 or 16 bits and save memory but not compute.

A plain-language summary

A trained AI model stores millions or billions of numbers with many decimal places. Post-training quantization rounds those numbers to a much shorter form after training is finished, much like saving a photo at a lower but still good quality setting.

The model becomes several times smaller and faster, so it can run on a phone or a small device instead of a distant server. The cost is a slight loss of precision, which engineers measure carefully. The key idea is simple: keep the learned knowledge, but store it with fewer digits.

An example from practice

A team building a smart doorbell needs a person-detection model to run on a low-power vision chip with an INT8 accelerator. The float model is about 14 MB and reaches 91.2% accuracy on its validation set, but the chip has only a few megabytes of fast memory available.

The engineers apply static INT8 post-training quantization with per-channel weight scales and calibrate it on 500 frames recorded by real doorbells at day, dusk and night. The quantized model shrinks to about 3.6 MB, runs fully on the accelerator and reaches 90.6% accuracy. Inference time falls from roughly 120 ms on the CPU to under 20 ms.

An earlier attempt calibrated on studio photos had lost several points of accuracy at night. Representative calibration data closed most of that gap, and the team shipped without needing quantization-aware training.