4-bit quantization is a model compression technique that represents neural network weights, and sometimes activations, with 4-bit integers instead of 16-bit or 32-bit floating-point numbers. In Edge AI, it is the main reason a language model with several billion parameters can run on a laptop, a smartphone or a single consumer GPU rather than in a data center.
Because each value has only 16 possible levels, the method depends on careful scaling, grouping and calibration to keep accuracy close to the original model. The approach is also called INT4 quantization, although several 4-bit formats exist, including floating-point variants such as NF4 and FP4.
How 4-bit quantization maps weights to integers
4-bit quantization works by mapping a range of real-valued weights onto a small grid of integer codes and storing a scale factor that converts the codes back to approximate real values. For a symmetric scheme, the scale is usually the largest absolute weight in a block divided by 7, and each weight becomes round(w / scale), clipped to the range -8 to 7. An asymmetric scheme adds a zero point so that the codes 0 to 15 cover a range that is not centered on zero.
Applying one scale to an entire tensor would destroy accuracy, because a single large outlier stretches the grid and leaves most weights rounded to zero. Practical schemes therefore use group-wise quantization: weights are split into groups of 32, 64 or 128 values, each with its own scale and sometimes its own zero point. Smaller groups improve accuracy but add metadata, so the effective storage cost is closer to 4.1 to 4.5 bits per weight than to a clean 4 bits.
The table below shows how the storage cost of a 7-billion-parameter model changes with precision, counting weights only.
| Precision | Bits per weight | Weight storage | Typical hardware |
|---|---|---|---|
| FP32 | 32 | About 28 GB | Server GPU or CPU with large RAM |
| FP16 or BF16 | 16 | About 14 GB | Data center or high-end GPU |
| INT8 | 8 | About 7 GB | Workstation GPU, high-memory PC |
| INT4 with groups of 128 | About 4.25 | About 3.7 GB | Laptop, flagship phone, consumer GPU |
Formats and variants of 4-bit quantization
Several 4-bit formats exist, and they differ in where the 16 levels are placed and in which tensors are quantized. Uniform integer grids are simple and fast on integer hardware, while non-uniform grids follow the bell-shaped distribution of trained weights more closely and lose less information.
The most common variants and their typical uses are summarized below.
| Format | Level spacing | Main use | Notable property |
|---|---|---|---|
| INT4 symmetric | Uniform, centered on zero | Weight-only inference kernels | Simplest dequantization |
| INT4 asymmetric | Uniform with zero point | Weights with skewed ranges | Better fit for shifted distributions |
| NF4 | Normal-distribution quantiles | Fine-tuning with frozen quantized weights | Information-efficient for Gaussian-like weights |
| FP4 (E2M1) | Exponential, sign plus exponent and mantissa | Newer accelerators with native FP4 | Wide dynamic range per level |
| W4A16 | 4-bit weights, 16-bit activations | Local LLM inference | Most widely deployed configuration |
| W4A4 | 4-bit weights and activations | Throughput-oriented accelerators | Hardest to keep accurate |
In practice, most edge deployments use weight-only quantization (W4A16): weights are stored in 4 bits and expanded to 16-bit values inside the matrix multiplication kernel. This configuration targets the memory bandwidth bottleneck of autoregressive token generation, where reading the weights dominates the cost of each step.
Methods that preserve accuracy at 4 bits
Naive round-to-nearest (RTN) quantization often works acceptably at 8 bits but can noticeably degrade a model at 4 bits, so specialized post-training quantization algorithms are used. They rely on a small calibration set, typically a few hundred text samples, to decide how to round and scale each group of weights.
- GPTQ quantizes weights column by column and uses approximate second-order information to compensate for rounding errors in the remaining weights.
- AWQ identifies the small share of weight channels that matter most for activations and rescales them before rounding to protect them.
- SmoothQuant shifts activation outliers into the weights, which matters mainly when activations are also quantized.
- Quantization-aware training simulates 4-bit rounding during training or fine-tuning, which usually gives the best accuracy at a much higher compute cost.
Related techniques, such as per-channel quantization, mixed-precision quantization and quantization calibration, describe building blocks that these methods combine. Mixed precision is common in practice: sensitive layers such as embeddings or the output head may stay at 8 or 16 bits while the bulk of the model is 4-bit.
How 4-bit quantization compares with other compression options
4-bit quantization sits between 8-bit quantization, which is nearly lossless for most models, and extreme formats such as binarized neural networks, which usually require retraining. The comparison below assumes a modern transformer language model.
| Technique | Size reduction vs FP16 | Typical accuracy impact | Retraining needed |
|---|---|---|---|
| INT8 quantization | About 2x | Usually negligible | No |
| 4-bit quantization | About 3.5x to 4x | Small with GPTQ or AWQ | No, calibration only |
| 3-bit or 2-bit quantization | About 5x to 8x | Often significant | Usually helpful |
| Structured pruning | Varies, often 1.3x to 2x | Moderate | Usually yes |
| Knowledge distillation | Depends on student size | Depends on student capacity | Yes |
These techniques are complementary rather than exclusive. A distilled or pruned model can still be quantized to 4 bits, and the combination is a common path to a small model footprint on embedded hardware.
Limitations and common mistakes
The main trade-off of 4-bit quantization is a small but real loss of model quality in exchange for a large memory saving. The loss tends to be larger for small models, for tasks that require long reasoning chains or precise arithmetic, and for languages that were underrepresented in the calibration data.
| Symptom | Likely cause | Remedy |
|---|---|---|
| Perplexity rises sharply | Plain RTN with large groups | Use GPTQ or AWQ with group size 128 or smaller |
| Garbled output on one task | Calibration data unlike the task | Calibrate with representative samples |
| No speedup on device | Missing 4-bit kernels or delegate | Use a runtime with native INT4 support |
| Quality drop in small models | Too little redundancy to absorb error | Keep sensitive layers at 8 bits |
A frequent misconception is that 4-bit weights always make inference faster. Speed gains depend on kernel support: without optimized dequantization kernels or a suitable hardware delegate, the device may unpack weights inefficiently. Benchmarks should also measure task accuracy, not only perplexity, because perplexity can hide failures on specific skills.
Stripped to the essentials
An AI model is a huge list of numbers. Normally each number is stored with high precision, which takes considerable memory. 4-bit quantization rounds every number to one of only 16 steps, like replacing a detailed ruler with one that has just a few marks.
The model becomes about four times smaller, so it can fit on a phone or laptop. Clever rounding methods keep the answers almost as good as before. The cost is a small drop in quality, which is usually acceptable when the alternative is not running the model on the device at all.
Where it is applied
A legal services team wants a private assistant that summarizes contracts on staff laptops, without sending documents to a cloud service. The chosen open model has 8 billion parameters, which needs about 16 GB in FP16, more than the shared memory available on their 16 GB machines once the operating system and applications are running.
The engineers apply AWQ with group size 128, calibrating on a few hundred anonymized contract paragraphs. The weights shrink to about 4.5 GB, and the model runs in a local inference runtime at roughly 20 to 30 tokens per second. On an internal test set of summaries, reviewers rate the 4-bit output nearly indistinguishable from the FP16 version, and the documents never leave the device.