Mixed-precision quantization is a model compression method that stores and computes different parts of a neural network at different numerical bit widths, for example 8 bits for most layers, 4 bits for robust ones and 16 bits for a few sensitive ones. In Edge AI, it is one of the main ways to make a model small and fast enough for phones, PCs and embedded devices while keeping accuracy close to the original floating-point version.
The idea rests on a simple observation: layers are not equally sensitive to rounding error. Uniform quantization forces every tensor to the same precision, so the most fragile layer sets the limit for the whole model. A mixed-precision scheme spends bits where they protect accuracy and removes them where they only cost memory, bandwidth and energy.
How mixed-precision quantization allocates bits
Mixed-precision quantization works by measuring how much each layer, channel or tensor contributes to output error and then solving a bit allocation problem under a hardware budget. The budget is usually a target model size in megabytes, a latency limit on a specific chip or an energy cap per inference.
The process has three stages. First, a sensitivity analysis estimates the damage caused by lowering the precision of each candidate unit. Second, a search or optimization step chooses a precision for every unit so that total cost stays within the budget and estimated error is minimal. Third, the network is quantized with the chosen configuration, calibrated, and often fine-tuned to recover the remaining loss.
The following table lists the sensitivity signals most often used to rank layers before bits are assigned.
| Signal | What it measures | Cost to compute |
|---|---|---|
| Hessian trace | Curvature of the loss around the weights | High, needs second-order estimates |
| Quantization noise ratio | Output error relative to signal per layer | Low, one calibration pass |
| Accuracy drop probing | Task metric after quantizing one layer | Medium to high, many evaluations |
| KL divergence | Shift in activation distribution | Low, calibration statistics only |
| Learned bit width | Precision trained as a differentiable parameter | High, requires training |
Granularity and precision formats
Mixed precision can be applied at several levels of granularity, and the right level depends on what the target accelerator can execute efficiently. Finer granularity captures more accuracy but adds metadata, kernel variety and scheduling overhead.
- Per-layer precision, the most widely supported form, where each convolution or linear layer receives one bit width.
- Per-tensor split between weights and activations, such as 4-bit weights with 8-bit activations (often written W4A8).
- Per-channel or per-group precision, which keeps a small set of outlier channels at higher precision.
- Per-block precision in large language models, where attention projections and feed-forward blocks receive different formats.
The table below summarizes the formats that typically appear together in one mixed-precision model and where each one is usually placed.
| Format | Size per value | Typical placement |
|---|---|---|
| FP16 or BF16 | 16 bits | First and last layers, normalization, softmax |
| INT8 | 8 bits | Most convolutions and activations |
| INT4 | 4 bits | Large weight matrices in robust layers |
| INT2 or binary | 1 to 2 bits | Rare, only highly redundant layers |
| FP8 (E4M3, E5M2) | 8 bits | Accelerators with native FP8 units |
A common rule of thumb is that the first convolution, the classifier head and embedding tables stay at higher precision, because errors there either propagate through every later layer or directly distort the final scores.
How it compares with related compression methods
Mixed-precision quantization is a policy about bit widths, not a separate training regime, so it combines with other Edge AI compression techniques rather than replacing them. It can be applied after training or during training, and it often sits alongside pruning or weight clustering in the same deployment pipeline.
| Technique | What it changes | Relation to mixed precision |
|---|---|---|
| Uniform INT8 quantization | All tensors to one 8-bit format | Baseline that mixed precision improves on |
| Post-training quantization | Converts a trained model with calibration | Can host a mixed bit allocation |
| Quantization-aware training | Simulates rounding during training | Recovers accuracy after allocation |
| Weight-only quantization | Compresses weights, keeps activations wide | Often mixed across weight groups |
| Neural network pruning | Removes weights or structures | Complementary, reduces count not width |
Methods such as GPTQ and AWQ for language models frequently produce implicitly mixed results: most weights land at 4 bits while salient channels or outliers are protected at higher precision or with separate scales.
Choosing a configuration for target hardware
The best mixed-precision configuration is the one the target hardware actually accelerates, because a theoretically smaller model can run slower if its formats fall back to generic kernels. A 3-bit layer, for instance, saves memory on paper but may be unpacked to 8 bits at runtime on a mobile NPU.
Practitioners therefore use hardware-aware search, measuring real latency per layer and per format on the device or with an accurate cost model. The decision table below reflects common deployment situations.
| Situation | Recommended strategy | Reason |
|---|---|---|
| Microcontroller with INT8 only | INT8 everywhere, INT16 for sensitive activations | No sub-byte kernels available |
| Phone NPU with INT4 and INT8 | INT4 weights in large layers, INT8 elsewhere | Native support for both formats |
| Local LLM on a laptop | 4-bit weight groups, higher precision for embeddings | Memory bandwidth dominates speed |
| Strict accuracy target | Allocation plus quantization-aware fine-tuning | Closes the last accuracy gap |
| Tight engineering schedule | Post-training allocation by noise ratio | Fast, no retraining needed |
A sound workflow keeps a floating-point reference model, reports accuracy and latency for every candidate configuration, and freezes the chosen bit map in version control together with the calibration data.
Limitations and common mistakes
The main weakness of mixed-precision quantization is that sensitivity is not additive: quantizing two layers together can hurt more than the sum of quantizing each alone. Greedy allocations based on single-layer probing can therefore miss the true optimum.
- Ignoring runtime support produces configurations that convert formats between layers and lose the expected speedup.
- Too small a calibration set gives unstable ranges, so sensitivity scores change between runs.
- Optimizing model size only overlooks that activation memory and data movement often dominate energy use.
- Skipping per-class evaluation hides accuracy loss concentrated in rare classes or long-tail inputs.
Search cost is another limitation. Exhaustive search grows exponentially with the number of layers, so real tools rely on integer linear programming, reinforcement learning or differentiable search to find a good configuration in hours rather than weeks.
A non-technical view
A neural network stores millions of numbers, and each number takes space. Making every number shorter saves space, but some parts of the network need more detail than others to keep giving correct answers.
Mixed-precision quantization is like packing a suitcase with a weight limit. Fragile items get padding, while sturdy items are packed tightly. In the same way, sensitive parts of the model keep longer numbers and robust parts use very short ones. The result is a model that fits on a small device and still behaves almost like the full version.
A concrete example
An engineering team builds a smart doorbell that must detect people and packages on a low-power camera chip with a small neural accelerator. Their detection model in 32-bit floating point is about 24 MB, too large for the chip’s on-board memory, and uniform 4-bit quantization reduces accuracy by roughly 6 percentage points.
The team runs a sensitivity analysis with a few hundred calibration images. It keeps the first convolution and detection head at 8 bits, moves the large middle layers to 4 bits, and fine-tunes for two epochs with quantization-aware training. The final model is under 5 MB, runs within the frame-rate target, and loses less than one percentage point of accuracy, which allows the doorbell to process video locally without sending it to the cloud.