Per-channel quantization

On this page 7

Per-channel quantization is a quantization scheme in which each output channel of a weight tensor receives its own scale factor (and optionally its own zero point), instead of one shared scale for the whole tensor. In Edge AI, it is the default way to compress convolutional and linear layers to 8-bit or 4-bit integers while keeping accuracy close to the floating-point original.

The idea is simple but decisive: neural network weights vary widely in magnitude from one filter to the next, so a single scale wastes precision on small channels. By fitting the integer grid to each channel’s actual range, per-channel quantization removes most of the accuracy loss caused by outlier channels at almost no runtime cost.

How per-channel quantization maps weights to integers

Per-channel quantization applies the standard affine mapping q = round(w / s) + z separately to every slice of a weight tensor along its output channel axis. For a convolution kernel with shape [out, in, h, w], there is one scale s per output filter; for a fully connected layer, one scale per output neuron (each row of the weight matrix).

Most deployments use symmetric quantization for weights, which fixes the zero point at 0 and sets s = max(|w|) / 127 for INT8. Symmetric weights simplify the integer kernel because the cross terms that involve a weight zero point disappear. Activations, by contrast, usually stay per-tensor with an asymmetric zero point, because their ranges change with every input.

The table below shows how the granularity choice changes the number of stored parameters and the precision available for each weight.

GranularityScales storedPrecision per weightTypical use
Per-tensorOne per tensorLowest when ranges differActivations, simple accelerators
Per-channelOne per output channelHigh for most layersINT8 weights in CNNs and transformers
Per-groupOne per block of 32-128 weightsHighest of the three4-bit weight-only LLM quantization
Quantization granularity options for a weight tensor

Why channel-wise scales preserve accuracy

The core problem that per-channel quantization solves is range mismatch between channels. In a trained network, one filter may hold weights within ±0.02 while another reaches ±1.5. With a single per-tensor scale set by the largest value, the small filter is mapped onto only two or three integer levels, which effectively erases its information.

Range mismatch is especially severe in depthwise separable convolutions, used heavily in MobileNet. Each depthwise filter sees only one input channel, so filter magnitudes can differ by two orders of magnitude. Per-tensor INT8 quantization of MobileNetV2 can lose several percentage points of top-1 accuracy or collapse entirely without corrective steps, while per-channel quantization typically keeps the drop below about one point.

Batch normalization folding makes the problem worse and the solution more valuable. When batch normalization parameters are merged into the preceding convolution before export, each output channel is multiplied by a different factor, which widens the spread of channel ranges. Per-channel scales absorb that spread naturally.

The comparison below summarizes the typical outcomes reported for common model families; exact numbers depend on the calibration data, the export toolchain and the target runtime.

Model familyPer-tensor weightsPer-channel weightsMain reason
Standard ResNetSmall lossNegligible lossModerate channel spread
MobileNet with depthwise layersLarge or severe lossSmall lossExtreme filter range differences
Transformer linear layersModerate lossSmall lossUneven output neuron norms
Typical effect of weight granularity on INT8 post-training accuracy

Runtime and hardware support

Per-channel weight scales add very little inference cost because they can be applied after the integer accumulation. An INT8 matrix multiply accumulates products in 32-bit integers, and the requantization step then multiplies each output channel by its combined scale, which is a vector instead of a scalar. The extra memory is one 32-bit value per output channel, usually far below 0.1% of the layer size.

Support is broad but not universal. The TensorFlow Lite integer specification requires per-axis symmetric weights for convolutions, ONNX expresses the scheme through the axis attribute of QuantizeLinear and DequantizeLinear, and PyTorch provides per-channel observers in its quantization workflows. Some fixed-function NPUs and older DSPs accept only per-tensor scales, so the edge inference runtime may fall back to slower paths or reject the model.

The table below lists the practical checks to perform before relying on per-channel weights on a given device.

CheckWhat to verifyConsequence if missing
Operator coverageEvery quantized layer accepts per-axis scalesLayers fall back to CPU or float
Hardware delegateNPU or DSP driver supports vector scalesModel partitioned across processors
Requantization pathFixed-point multipliers per channelRounding errors or failed conversion
Model conversion toolExporter keeps the axis attributeSilent conversion to per-tensor
Deployment checks for per-channel weights

How it relates to other quantization techniques

Per-channel quantization is a granularity choice, not a full workflow, so it combines with other Edge AI compression methods rather than competing with them. It is used inside post-training quantization, inside quantization-aware training and inside most INT8 quantization pipelines for weights.

  • Post-training quantization uses per-channel weights to reach acceptable INT8 accuracy without retraining, often after a short quantization calibration pass for activations.
  • Quantization-aware training simulates per-channel rounding during fine-tuning, which recovers most of the remaining error at the cost of training time.
  • Per-group quantization splits each channel further into blocks, which 4-bit quantization methods such as GPTQ and AWQ need because a single channel scale is too coarse at 16 integer levels.
  • SmoothQuant addresses the opposite problem, outliers in activation channels, by moving scale from activations into weights, which then rely on per-channel scales to absorb the change.

Limitations and common mistakes

Per-channel quantization does not fix outliers within a single channel. When one weight in a row is far larger than the rest, the whole row still loses precision, and clipping, per-group scales or weight equalization are needed. The scheme also does nothing for activations, which often dominate the error in transformer models.

The most frequent practical failure is choosing the wrong axis. Scales must follow the output channel dimension; quantizing along the input dimension cannot be folded into the requantization step and produces either wrong results or an unsupported graph.

SymptomLikely causeFix
Accuracy still drops sharplyActivation outliers dominateBetter calibration or SmoothQuant
Model runs slower on the NPUDelegate lacks per-axis supportCheck operator support or use per-tensor
Converted model gives wrong outputsScales applied on the input axisQuantize along the output channel axis
Tiny channels quantize to zeroDead or nearly dead filtersPrune them or keep a minimum scale
Common symptoms and fixes

A final caution concerns reporting. Accuracy figures for quantized models are only comparable when the granularity is stated, because a per-channel result and a per-tensor result for the same bit width can differ substantially.

A non-technical view

Shrinking an AI model means storing its numbers with fewer digits. The trick is choosing the measuring scale. If one ruler must measure both an ant and a building, the ant ends up as zero. Per-channel quantization gives every group of numbers its own ruler, so small details are not rounded away.

Because each ruler is just one extra number, the model stays almost as small and fast. The result is a compact model that behaves almost like the original, which is exactly what phones and small devices need.

A concrete example

An engineering team building a smart doorbell wants to run a MobileNetV2-based person detector on a low-power microcontroller with an integer-only accelerator. The float model reaches about 92% detection accuracy on their test set, but it needs roughly 14 MB, while the device has about 4 MB of usable flash.

Their first INT8 conversion with per-tensor weights shrinks the model by roughly four times but drops accuracy to around 80%, because several depthwise filters collapse to a few integer values. Switching the exporter to symmetric per-channel weights and calibrating activations on 500 typical doorway images brings accuracy back to about 91%.

The extra scales add only a few kilobytes. The team confirms that the accelerator driver supports per-axis scales, so inference stays at under 60 milliseconds per frame, and the doorbell ships with on-device detection and no cloud dependency.