SmoothQuant is a training-free post-training quantization method that makes 8-bit weights and 8-bit activations (W8A8) practical for large transformer models by mathematically moving the difficulty of quantizing activations into the weights. In Edge AI and in data-center inference alike, it allows a model to use integer matrix multiplication throughout its linear layers, which reduces memory traffic and speeds up computation without retraining.
The method was introduced in 2022 by researchers from MIT and NVIDIA and presented at ICML 2023. Its central observation is that in large language models, weights are easy to quantize but activations are not, because a few activation channels carry systematic outliers that are many times larger than the rest.
Why activation outliers break 8-bit quantization
Uniform INT8 quantization maps a range of real values onto 256 integer levels using a single scale factor. When one value in a tensor is far larger than the others, the scale must stretch to cover it, and the remaining values collapse into a handful of levels. The result is large rounding error for ordinary values.
In transformer models beyond roughly 6 to 7 billion parameters, activations entering linear layers develop outliers concentrated in a small, fixed set of channels. These channels can be up to about 100 times larger than typical values. Because the outliers sit in the same channels across tokens, per-token activation scaling cannot remove them, and naive W8A8 quantization causes sharp accuracy loss.
The table below contrasts how weights and activations behave under quantization.
| Property | Weights | Activations |
|---|---|---|
| Value distribution | Fairly uniform, few extremes | Heavy outliers in specific channels |
| Known in advance | Yes, fixed after training | No, depend on each input |
| INT8 error without help | Low | High in large models |
| Preferred granularity | Per output channel | Per tensor or per token |
How SmoothQuant migrates quantization difficulty
SmoothQuant applies a per-channel smoothing factor that divides each activation channel by a scale s and multiplies the matching weight row by the same scale. Because Y = (X diag(s)^-1)(diag(s) W), the layer output is mathematically unchanged, but the activation outliers shrink while the weights grow slightly.
The scale for channel j is computed as s_j = max|X_j|^α / max|W_j|^(1-α), where the migration strength α controls how much difficulty moves to the weights. The activation maxima come from a short calibration run of a few hundred samples. The division is then folded offline into the preceding LayerNorm or linear layer, so smoothing adds no runtime operations.
The table below shows how the migration strength affects the outcome.
| Migration strength α | Activations after smoothing | Weights after smoothing | Typical use |
|---|---|---|---|
| 0.0 | Unchanged, outliers remain | Unchanged | Baseline, no smoothing |
| 0.5 | Outliers strongly reduced | Moderately harder to quantize | Default for many models |
| 0.75 to 0.85 | Very flat | Noticeably harder to quantize | Models with extreme outliers |
| 1.0 | Nearly uniform | Absorb all difficulty, accuracy suffers | Rarely useful |
SmoothQuant compared with other quantization methods
SmoothQuant belongs to the family of post-training quantization techniques, but it targets a different bottleneck than weight-only methods such as GPTQ and AWQ. Those methods compress weights to 4 bits and keep activations in floating point, which saves memory but still uses floating-point arithmetic.
| Method | Precision | Main benefit | Main limitation |
|---|---|---|---|
| SmoothQuant | W8A8 | Integer compute for all linear layers | Needs INT8 kernel support for good speedup |
| LLM.int8() | Mixed INT8 and FP16 | Keeps outlier channels in FP16 | Slower due to mixed decomposition |
| GPTQ | W4 or W3, FP16 activations | Strong memory reduction | No integer activation compute |
| AWQ | W4, FP16 activations | Protects salient weight channels | Same compute limitation as GPTQ |
| Quantization-aware training | Flexible | Highest accuracy at low bit widths | Requires costly retraining |
The original evaluation reported near-lossless accuracy on models such as OPT-175B, BLOOM-176B and GLM-130B, with up to about 1.5 times faster inference and roughly half the memory compared with FP16. Exact gains depend on hardware, batch size and kernel quality.
Applying SmoothQuant on edge and server hardware
SmoothQuant pays off where INT8 matrix units are available: NVIDIA tensor cores, recent CPU vector extensions, and many neural processing units in phones and PCs. On devices that only accelerate floating point, the speedup largely disappears even though accuracy is preserved.
- Collect calibration data that resembles real prompts, typically a few hundred sequences.
- Record per-channel activation maxima for every linear layer input.
- Choose α per model, often by a small sweep against a validation set.
- Fold the scales into preceding layers and quantize weights per channel.
- Select activation granularity: per-tensor static, per-tensor dynamic or per-token dynamic.
The granularity choice trades speed for accuracy, as the table below summarizes.
| Variant | Scale computed | Speed | Accuracy |
|---|---|---|---|
| Per-tensor static | Once, during calibration | Fastest | Lowest of the three |
| Per-tensor dynamic | Per batch at runtime | Fast | Better |
| Per-token dynamic | For each token at runtime | Slightly slower | Best |
Limitations and practical pitfalls
- Calibration mismatch: scales learned on unrepresentative text can clip outliers seen in production.
- Not a 4-bit method: at W4A4, SmoothQuant alone usually loses too much accuracy and is combined with rotation-based or mixed-precision techniques.
- Uncovered operations: attention softmax, normalization and some matrix products may stay in higher precision.
- Kernel dependence: without optimized INT8 kernels, quantized models can run slower than FP16.
For small edge models below a few billion parameters, outliers are often mild, and plain INT8 quantization may already work. SmoothQuant is most valuable when standard calibration fails because of channel outliers.
Explained without jargon
Shrinking a model to 8-bit numbers is like recording a song on equipment with limited volume range. If one instrument is extremely loud, everything else must be turned down so far that it becomes hard to hear. SmoothQuant turns the loud parts down and compensates elsewhere, so the whole song fits the range.
The trick is that the compensation happens in the model’s fixed settings, which handle it easily. The answers stay the same, but the model needs less memory and can use faster arithmetic.
A typical use case
A company deploys a 13-billion-parameter language model on an on-premises inference server with a single data-center GPU that supports INT8 tensor operations. In FP16, the model leaves little memory for longer prompts and handles few concurrent users.
The engineering team runs SmoothQuant with 512 calibration samples drawn from real support tickets, tests α values from 0.5 to 0.8, and selects 0.6 with per-token dynamic activations. Weights and activations then run in INT8 across all linear layers.
The result is roughly half the weight memory, higher throughput per request batch, and a benchmark score within a fraction of a point of the FP16 baseline. The same workflow later prepares a smaller model for laptops with INT8-capable neural processing units.