AWQ

Also known as: Activation-aware weight quantization

On this page 7

AWQ (activation-aware weight quantization) is a post-training, weight-only quantization method that compresses the weights of large language models to low precision, typically 4 bits, by using activation statistics to decide which weight channels matter most and protecting them with per-channel scaling. In Edge AI, it is one of the standard ways to make a model small enough to run on a laptop GPU, a phone or an embedded accelerator instead of in the cloud.

The method was introduced in 2023 by researchers associated with MIT and quickly became a default format in open-source inference engines. Its central insight is that weight importance is revealed by activations, not by weights themselves, and that a simple, hardware-friendly scaling trick can preserve those important weights without keeping any of them in higher precision.

How AWQ protects salient weights

AWQ works by identifying salient weight channels from activation magnitudes and scaling those channels up before rounding, so that they lose less relative precision. The authors observed that only a small fraction of weights, roughly 0.1-1%, has a disproportionate effect on output quality, and that these weights belong to the input channels that receive large activation values.

Keeping that small fraction in FP16 restores most of the lost accuracy, but such mixed-precision storage is awkward for hardware because it breaks the regular memory layout that fast kernels rely on. AWQ instead applies an equivalent transformation: it multiplies each weight input channel by a factor s and divides the corresponding activation by the same s. Mathematically the layer output is unchanged, yet the scaled weights are rounded with a smaller relative error.

The scaling factor is chosen per channel as the average activation magnitude raised to an exponent alpha, and alpha is found by a simple grid search between 0 and 1 that minimizes the layer output error on calibration data. The inverse scale is usually folded into the preceding operator, such as a normalization layer or the previous linear layer, so inference pays no extra cost.

The process runs layer by layer and consists of a few well-defined steps, summarized below.

StepWhat happensPurpose
Calibration passA small text sample flows through the modelCollect per-channel activation magnitudes
Scale searchGrid search over alpha for each layerFind scales that minimize output error
Weight clipping searchTest reduced maximum ranges per groupLimit the influence of outlier weights
Group quantizationRound scaled weights to 4-bit integersStore weights with per-group scale and zero point
Scale foldingMerge inverse scales into earlier operatorsKeep inference math unchanged and fast
Main steps of the AWQ procedure

Key parameters of an AWQ model

An AWQ checkpoint is defined mainly by its bit width, group size and zero-point setting, and these choices control the balance between size and accuracy. The most common configuration is W4A16: 4-bit weights with 16-bit activations, which means the matrix multiplication still runs in FP16 after the weights are dequantized inside the kernel.

Group quantization assigns a separate scale and zero point to each block of consecutive weights. A group size of 128 is the usual default; smaller groups improve accuracy but add metadata, while larger groups save memory at a measurable quality cost.

ParameterCommon valueEffect of changing it
Weight bit width4 bits3 bits saves memory but degrades quality noticeably
Group size12864 improves accuracy and adds metadata overhead
Zero pointEnabled (asymmetric)Symmetric mode is simpler but less precise
Calibration samplesAbout 128 sequencesMore samples rarely change results much
Activation precisionFP16 or BF16Unchanged by AWQ, which is weight-only
Typical AWQ settings and their effects

With 4-bit weights and a group size of 128, the effective storage cost is about 4.25 bits per weight once scales and zero points are included. A 7-billion-parameter model therefore shrinks from about 14 GB in FP16 to roughly 4 GB, which fits on consumer GPUs and high-end mobile devices.

AWQ compared with other quantization methods

AWQ belongs to the family of post-training quantization techniques and is most often compared with GPTQ, SmoothQuant and simple round-to-nearest quantization. The main difference is that AWQ does not use backpropagation or weight reconstruction; it only searches for scales, which makes it fast and less prone to overfitting the calibration set.

MethodCore ideaTypical precisionCalibration dependence
AWQActivation-guided per-channel scalingW4A16Low
GPTQSecond-order error compensation per columnW4A16 or W3A16Moderate
SmoothQuantMoves activation outliers into weightsW8A8Low
Round-to-nearestPlain rounding without any searchW8 or W4None
Quantization-aware trainingLearns quantized weights during trainingW4 or lowerNeeds training data
How AWQ differs from related methods

Compared with GPTQ, AWQ generally achieves similar or slightly better accuracy at 4 bits and tends to generalize better across domains, for example to code or instruction-tuned behavior that the calibration text did not cover. GPTQ can be stronger at 3 bits in some benchmarks, so the gap depends on model family and evaluation. SmoothQuant targets a different goal, 8-bit activation quantization, and the two ideas can be combined.

Using AWQ in edge deployment

AWQ is most useful when memory bandwidth, not compute, limits inference speed, which is the normal case for token-by-token generation in language models. Loading 4-bit weights instead of 16-bit weights cuts the data moved per token by about four times, so decoding speedups of roughly 2-3x over FP16 are common on GPUs with optimized kernels.

Quantizing a model with AWQ usually takes minutes to a few hours on a single GPU, depending on model size. The resulting checkpoints are supported by widely used open-source serving engines and model libraries, which makes AWQ a practical choice for local LLM and hybrid inference setups.

SituationRecommendationReason
LLM chat on a consumer GPUAWQ at 4 bitsLarge memory savings with small quality loss
Throughput-bound server batchingConsider W8A8 methodsCompute dominates at large batch sizes
Microcontroller or TinyML targetUse INT8 full quantizationAWQ kernels target GPUs and NPUs
Domain-specific modelCalibrate on in-domain textActivation statistics should match real use
Extreme compression below 4 bitsEvaluate carefully or use trainingPost-training error grows quickly
When AWQ is a good choice

Limitations and common mistakes

AWQ reduces model size effectively, but it only quantizes weights, so activation memory and the KV cache remain in 16-bit precision unless other techniques are added. Its benefits also depend on dedicated dequantization kernels; without them, a 4-bit model can run slower than the FP16 original.

  • Assuming zero accuracy loss: perplexity rises slightly, and reasoning or long-context tasks can degrade more than headline benchmarks show.
  • Ignoring hardware support: some NPUs and mobile runtimes expect other 4-bit formats, so model conversion may be required.
  • Using unrepresentative calibration data: although AWQ is robust, extreme domain mismatch can still mis-rank salient channels.
  • Expecting faster prefill: compute-bound prompt processing gains little, because the matrix multiplications still run in FP16.

A plain-language summary

AWQ is a way to shrink an AI language model so it fits on smaller devices. It stores each number in the model with far fewer digits, which normally makes the model less accurate.

The trick is to notice that a few numbers matter much more than the rest. AWQ finds them by watching which inputs are usually large, then gives those numbers extra room before rounding. It is like packing a suitcase: fragile items get padding, and everything else is folded tightly. The model becomes about four times smaller while answering almost as well as before.

An example from practice

A small engineering firm wants an internal assistant that answers questions about its design documents without sending data to a cloud service. Its target machine is an office workstation with a 24 GB consumer GPU, while the chosen 13-billion-parameter model needs about 26 GB in FP16.

The team quantizes the model with AWQ at 4 bits and group size 128, calibrating on about 128 passages drawn from its own technical documentation. The process takes under an hour. The resulting model occupies roughly 7 GB of GPU memory, leaving room for a long KV cache and a retrieval index.

On the firm’s internal test questions, answer quality is judged nearly identical to the original, and generation runs at a comfortable reading speed. The outcome is a fully local assistant on existing hardware, with sensitive documents never leaving the building.