Weight-only quantization is a model compression method that stores the weights of a neural network in low-precision integers, typically 8, 4 or 3 bits, while activations remain in 16-bit floating point. In edge AI, it is the most common way to make large language models fit into the memory of phones, laptops and embedded boards without retraining them.
The method targets the resource that limits local inference most: memory capacity and memory bandwidth. Because the weights are converted back to higher precision just before each multiplication, the arithmetic itself stays in floating point, and the accuracy loss is usually small at 8 bits and moderate at 4 bits.
How weight-only quantization works
Weight-only quantization maps each floating-point weight to a small integer plus a shared scale factor, and sometimes a zero point. A weight is recovered approximately as the integer multiplied by the scale, so a 4-bit weight can take only 16 distinct levels within its group.
Scales are shared by blocks of weights rather than by the whole tensor. Per-channel scaling assigns one scale to each output channel, while group-wise scaling assigns one scale to every 32, 64 or 128 consecutive weights. Smaller groups track local value ranges better but add storage overhead for the scales.
At inference time, an optimized kernel loads the packed integers, dequantizes them on the fly into registers and multiplies them with 16-bit activations. Since the weights never travel through memory at full size, the kernel moves two to four times fewer bytes than an FP16 kernel, which is the source of most speedups.
The table below lists the main parameters that define a weight-only scheme and what each one affects.
| Parameter | Typical values | What it affects |
|---|---|---|
| Bit width | 8, 4, 3 or 2 bits | Model size and accuracy loss |
| Group size | 32 to 128 weights, or per channel | Accuracy versus scale overhead |
| Symmetric or asymmetric | With or without a zero point | Fit to skewed weight ranges |
| Data format | Integer, NF4 or FP4 | Distribution of quantization levels |
| Calibration data | None, or 128 to 512 samples | Error correction in advanced methods |
| Excluded layers | Embeddings, output head | Accuracy of sensitive layers |
Why it suits edge inference
Weight-only quantization suits edge devices because token-by-token generation is memory-bound. When a language model generates text for one user, each step reads every weight once but performs relatively few operations per byte, so the processor waits on memory rather than on arithmetic.
Reducing weight size therefore speeds up generation almost in proportion to the bytes saved, while activation precision barely matters for this workload. A 7-billion-parameter model needs about 14 GB in FP16 but only about 4 GB at 4 bits including scales, which moves it from server-class hardware to a mainstream laptop or a high-end phone.
| Precision | Weight memory | Typical target device |
|---|---|---|
| FP16 | About 14 GB | Workstation GPU |
| INT8 weight-only | About 7 GB | Laptop with 16 GB memory |
| 4-bit weight-only | About 3.5 to 4 GB | Thin laptop or high-end phone |
| 3-bit weight-only | About 2.7 to 3 GB | Memory-constrained phone |
| 2-bit weight-only | About 2 GB | Experimental, notable quality loss |
The benefit is smaller for batched or prompt-processing workloads, which are compute-bound. There, the dequantization step adds work, and schemes that also quantize activations can run faster on hardware with integer matrix units.
Weight-only quantization compared with related methods
Weight-only quantization is one point in a wider design space of post-training quantization methods. It differs from full integer schemes in what is quantized and from training-based approaches in how the model is prepared.
| Method | What is quantized | Main trade-off |
|---|---|---|
| Weight-only quantization | Weights only | Large memory savings, little compute gain |
| INT8 weight and activation quantization | Weights and activations | Faster integer math, harder outlier handling |
| Dynamic quantization | Weights, activations at runtime | Simple setup, runtime scaling cost |
| Quantization-aware training | Simulated during training | Best accuracy, needs training compute |
| Mixed-precision quantization | Different bits per layer | Better accuracy per byte, complex tuning |
Several well-known algorithms implement weight-only quantization. GPTQ uses second-order information from calibration data to compensate rounding errors column by column, while AWQ protects the small fraction of weights that multiply large activations by rescaling channels before rounding. Simple round-to-nearest quantization needs no data and works well at 8 bits but degrades faster at 4 bits.
Activation-aware methods such as SmoothQuant belong to the weight-and-activation family: they shift outlier magnitude from activations into weights so that both can be stored as INT8.
Applying weight-only quantization in practice
A practical workflow starts by choosing the bit width from the memory budget, then selecting an algorithm and group size, and finally measuring quality on the target task. Runtimes for local inference usually ship ready-made formats, so the choice is often between published variants of the same model.
- Estimate memory: weights plus the key-value cache plus runtime overhead must fit with headroom.
- Pick 8 bits when quality is critical and memory allows, 4 bits with groups of 32 to 128 as the common default.
- Use a calibration-based method such as GPTQ or AWQ for 4 bits and below.
- Keep embeddings and the output layer at higher precision if accuracy drops.
- Evaluate perplexity and task-specific benchmarks, not only general scores.
Hardware support decides whether savings become speed. Efficient fused dequantization kernels exist for common GPUs, Apple silicon and mobile CPUs, but an unsupported format may fall back to slow code or be expanded to FP16 at load time, losing the memory advantage.
Limitations and common mistakes
The main limitation is that quality degrades sharply below about 4 bits, especially for smaller models, which have less redundancy than large ones. Reasoning, code generation and multilingual tasks often suffer before general chat quality shows a visible change.
| Problem | Cause | Remedy |
|---|---|---|
| Output becomes repetitive or incoherent | Bit width too low for model size | Move to 4 or 8 bits, smaller groups |
| No speedup despite smaller files | Missing optimized kernel | Use a runtime with native support |
| Out-of-memory errors persist | Key-value cache ignored in budget | Shorten context or quantize the cache |
| Benchmark fine, task quality poor | Calibration data unlike real inputs | Calibrate on domain samples |
Another common misconception is that weight-only quantization reduces energy and latency in every case. For long prompts and batched serving, compute dominates, so gains shrink and can even reverse when dequantization overhead is high.
A non-technical view
An AI model is mostly a huge list of numbers. Normally each number is written with high detail, a bit like storing every price to many decimal places. Weight-only quantization rounds those numbers to a much shorter form, so the whole list takes up a quarter or less of the space.
The device then expands each rounded number briefly while it calculates, much like unpacking a vacuum-sealed bag only when you need what is inside. The model becomes small enough to run on your own phone or laptop, with answers that are usually almost as good as the original.
A concrete example
A field-service company wants technicians to query repair manuals on rugged laptops with 16 GB of shared memory and no reliable internet connection. Its chosen 8-billion-parameter language model needs about 16 GB in FP16, which leaves no room for the operating system or the retrieval index.
The engineering team applies 4-bit weight-only quantization with AWQ and a group size of 128, calibrating on several hundred passages from the manuals themselves. The weights shrink to roughly 4.5 GB, and the embedding and output layers stay in 16-bit precision.
On a test set of 300 real technician questions, answer accuracy stays within a few percentage points of the full-precision model, while generation speed on the laptops reaches a comfortable reading pace. The company ships the quantized model in its next software update, giving technicians offline answers in the field.