Mixed-precision quantization

On this page 7

Mixed-precision quantization is a model compression method that stores and computes different parts of a neural network at different numerical bit widths, for example 8 bits for most layers, 4 bits for robust ones and 16 bits for a few sensitive ones. In Edge AI, it is one of the main ways to make a model small and fast enough for phones, PCs and embedded devices while keeping accuracy close to the original floating-point version.

The idea rests on a simple observation: layers are not equally sensitive to rounding error. Uniform quantization forces every tensor to the same precision, so the most fragile layer sets the limit for the whole model. A mixed-precision scheme spends bits where they protect accuracy and removes them where they only cost memory, bandwidth and energy.

How mixed-precision quantization allocates bits

Mixed-precision quantization works by measuring how much each layer, channel or tensor contributes to output error and then solving a bit allocation problem under a hardware budget. The budget is usually a target model size in megabytes, a latency limit on a specific chip or an energy cap per inference.

The process has three stages. First, a sensitivity analysis estimates the damage caused by lowering the precision of each candidate unit. Second, a search or optimization step chooses a precision for every unit so that total cost stays within the budget and estimated error is minimal. Third, the network is quantized with the chosen configuration, calibrated, and often fine-tuned to recover the remaining loss.

The following table lists the sensitivity signals most often used to rank layers before bits are assigned.

SignalWhat it measuresCost to compute
Hessian traceCurvature of the loss around the weightsHigh, needs second-order estimates
Quantization noise ratioOutput error relative to signal per layerLow, one calibration pass
Accuracy drop probingTask metric after quantizing one layerMedium to high, many evaluations
KL divergenceShift in activation distributionLow, calibration statistics only
Learned bit widthPrecision trained as a differentiable parameterHigh, requires training
Common sensitivity signals for bit allocation

Granularity and precision formats

Mixed precision can be applied at several levels of granularity, and the right level depends on what the target accelerator can execute efficiently. Finer granularity captures more accuracy but adds metadata, kernel variety and scheduling overhead.

  • Per-layer precision, the most widely supported form, where each convolution or linear layer receives one bit width.
  • Per-tensor split between weights and activations, such as 4-bit weights with 8-bit activations (often written W4A8).
  • Per-channel or per-group precision, which keeps a small set of outlier channels at higher precision.
  • Per-block precision in large language models, where attention projections and feed-forward blocks receive different formats.

The table below summarizes the formats that typically appear together in one mixed-precision model and where each one is usually placed.

FormatSize per valueTypical placement
FP16 or BF1616 bitsFirst and last layers, normalization, softmax
INT88 bitsMost convolutions and activations
INT44 bitsLarge weight matrices in robust layers
INT2 or binary1 to 2 bitsRare, only highly redundant layers
FP8 (E4M3, E5M2)8 bitsAccelerators with native FP8 units
Typical precision formats in a mixed configuration

A common rule of thumb is that the first convolution, the classifier head and embedding tables stay at higher precision, because errors there either propagate through every later layer or directly distort the final scores.

Mixed-precision quantization is a policy about bit widths, not a separate training regime, so it combines with other Edge AI compression techniques rather than replacing them. It can be applied after training or during training, and it often sits alongside pruning or weight clustering in the same deployment pipeline.

TechniqueWhat it changesRelation to mixed precision
Uniform INT8 quantizationAll tensors to one 8-bit formatBaseline that mixed precision improves on
Post-training quantizationConverts a trained model with calibrationCan host a mixed bit allocation
Quantization-aware trainingSimulates rounding during trainingRecovers accuracy after allocation
Weight-only quantizationCompresses weights, keeps activations wideOften mixed across weight groups
Neural network pruningRemoves weights or structuresComplementary, reduces count not width
Mixed precision and neighboring techniques

Methods such as GPTQ and AWQ for language models frequently produce implicitly mixed results: most weights land at 4 bits while salient channels or outliers are protected at higher precision or with separate scales.

Choosing a configuration for target hardware

The best mixed-precision configuration is the one the target hardware actually accelerates, because a theoretically smaller model can run slower if its formats fall back to generic kernels. A 3-bit layer, for instance, saves memory on paper but may be unpacked to 8 bits at runtime on a mobile NPU.

Practitioners therefore use hardware-aware search, measuring real latency per layer and per format on the device or with an accurate cost model. The decision table below reflects common deployment situations.

SituationRecommended strategyReason
Microcontroller with INT8 onlyINT8 everywhere, INT16 for sensitive activationsNo sub-byte kernels available
Phone NPU with INT4 and INT8INT4 weights in large layers, INT8 elsewhereNative support for both formats
Local LLM on a laptop4-bit weight groups, higher precision for embeddingsMemory bandwidth dominates speed
Strict accuracy targetAllocation plus quantization-aware fine-tuningCloses the last accuracy gap
Tight engineering schedulePost-training allocation by noise ratioFast, no retraining needed
Deployment situations and recommended strategies

A sound workflow keeps a floating-point reference model, reports accuracy and latency for every candidate configuration, and freezes the chosen bit map in version control together with the calibration data.

Limitations and common mistakes

The main weakness of mixed-precision quantization is that sensitivity is not additive: quantizing two layers together can hurt more than the sum of quantizing each alone. Greedy allocations based on single-layer probing can therefore miss the true optimum.

  • Ignoring runtime support produces configurations that convert formats between layers and lose the expected speedup.
  • Too small a calibration set gives unstable ranges, so sensitivity scores change between runs.
  • Optimizing model size only overlooks that activation memory and data movement often dominate energy use.
  • Skipping per-class evaluation hides accuracy loss concentrated in rare classes or long-tail inputs.

Search cost is another limitation. Exhaustive search grows exponentially with the number of layers, so real tools rely on integer linear programming, reinforcement learning or differentiable search to find a good configuration in hours rather than weeks.

A non-technical view

A neural network stores millions of numbers, and each number takes space. Making every number shorter saves space, but some parts of the network need more detail than others to keep giving correct answers.

Mixed-precision quantization is like packing a suitcase with a weight limit. Fragile items get padding, while sturdy items are packed tightly. In the same way, sensitive parts of the model keep longer numbers and robust parts use very short ones. The result is a model that fits on a small device and still behaves almost like the full version.

A concrete example

An engineering team builds a smart doorbell that must detect people and packages on a low-power camera chip with a small neural accelerator. Their detection model in 32-bit floating point is about 24 MB, too large for the chip’s on-board memory, and uniform 4-bit quantization reduces accuracy by roughly 6 percentage points.

The team runs a sensitivity analysis with a few hundred calibration images. It keeps the first convolution and detection head at 8 bits, moves the large middle layers to 4 bits, and fine-tunes for two epochs with quantization-aware training. The final model is under 5 MB, runs within the frame-rate target, and loses less than one percentage point of accuracy, which allows the doorbell to process video locally without sending it to the cloud.