4-bit quantization

Also known as: INT4 quantization

On this page 7

4-bit quantization is a model compression technique that represents neural network weights, and sometimes activations, with 4-bit integers instead of 16-bit or 32-bit floating-point numbers. In Edge AI, it is the main reason a language model with several billion parameters can run on a laptop, a smartphone or a single consumer GPU rather than in a data center.

Because each value has only 16 possible levels, the method depends on careful scaling, grouping and calibration to keep accuracy close to the original model. The approach is also called INT4 quantization, although several 4-bit formats exist, including floating-point variants such as NF4 and FP4.

How 4-bit quantization maps weights to integers

4-bit quantization works by mapping a range of real-valued weights onto a small grid of integer codes and storing a scale factor that converts the codes back to approximate real values. For a symmetric scheme, the scale is usually the largest absolute weight in a block divided by 7, and each weight becomes round(w / scale), clipped to the range -8 to 7. An asymmetric scheme adds a zero point so that the codes 0 to 15 cover a range that is not centered on zero.

Applying one scale to an entire tensor would destroy accuracy, because a single large outlier stretches the grid and leaves most weights rounded to zero. Practical schemes therefore use group-wise quantization: weights are split into groups of 32, 64 or 128 values, each with its own scale and sometimes its own zero point. Smaller groups improve accuracy but add metadata, so the effective storage cost is closer to 4.1 to 4.5 bits per weight than to a clean 4 bits.

The table below shows how the storage cost of a 7-billion-parameter model changes with precision, counting weights only.

PrecisionBits per weightWeight storageTypical hardware
FP3232About 28 GBServer GPU or CPU with large RAM
FP16 or BF1616About 14 GBData center or high-end GPU
INT88About 7 GBWorkstation GPU, high-memory PC
INT4 with groups of 128About 4.25About 3.7 GBLaptop, flagship phone, consumer GPU
Approximate weight storage for a 7-billion-parameter model

Formats and variants of 4-bit quantization

Several 4-bit formats exist, and they differ in where the 16 levels are placed and in which tensors are quantized. Uniform integer grids are simple and fast on integer hardware, while non-uniform grids follow the bell-shaped distribution of trained weights more closely and lose less information.

The most common variants and their typical uses are summarized below.

FormatLevel spacingMain useNotable property
INT4 symmetricUniform, centered on zeroWeight-only inference kernelsSimplest dequantization
INT4 asymmetricUniform with zero pointWeights with skewed rangesBetter fit for shifted distributions
NF4Normal-distribution quantilesFine-tuning with frozen quantized weightsInformation-efficient for Gaussian-like weights
FP4 (E2M1)Exponential, sign plus exponent and mantissaNewer accelerators with native FP4Wide dynamic range per level
W4A164-bit weights, 16-bit activationsLocal LLM inferenceMost widely deployed configuration
W4A44-bit weights and activationsThroughput-oriented acceleratorsHardest to keep accurate
Common 4-bit formats

In practice, most edge deployments use weight-only quantization (W4A16): weights are stored in 4 bits and expanded to 16-bit values inside the matrix multiplication kernel. This configuration targets the memory bandwidth bottleneck of autoregressive token generation, where reading the weights dominates the cost of each step.

Methods that preserve accuracy at 4 bits

Naive round-to-nearest (RTN) quantization often works acceptably at 8 bits but can noticeably degrade a model at 4 bits, so specialized post-training quantization algorithms are used. They rely on a small calibration set, typically a few hundred text samples, to decide how to round and scale each group of weights.

  • GPTQ quantizes weights column by column and uses approximate second-order information to compensate for rounding errors in the remaining weights.
  • AWQ identifies the small share of weight channels that matter most for activations and rescales them before rounding to protect them.
  • SmoothQuant shifts activation outliers into the weights, which matters mainly when activations are also quantized.
  • Quantization-aware training simulates 4-bit rounding during training or fine-tuning, which usually gives the best accuracy at a much higher compute cost.

Related techniques, such as per-channel quantization, mixed-precision quantization and quantization calibration, describe building blocks that these methods combine. Mixed precision is common in practice: sensitive layers such as embeddings or the output head may stay at 8 or 16 bits while the bulk of the model is 4-bit.

How 4-bit quantization compares with other compression options

4-bit quantization sits between 8-bit quantization, which is nearly lossless for most models, and extreme formats such as binarized neural networks, which usually require retraining. The comparison below assumes a modern transformer language model.

TechniqueSize reduction vs FP16Typical accuracy impactRetraining needed
INT8 quantizationAbout 2xUsually negligibleNo
4-bit quantizationAbout 3.5x to 4xSmall with GPTQ or AWQNo, calibration only
3-bit or 2-bit quantizationAbout 5x to 8xOften significantUsually helpful
Structured pruningVaries, often 1.3x to 2xModerateUsually yes
Knowledge distillationDepends on student sizeDepends on student capacityYes
Compression options compared

These techniques are complementary rather than exclusive. A distilled or pruned model can still be quantized to 4 bits, and the combination is a common path to a small model footprint on embedded hardware.

Limitations and common mistakes

The main trade-off of 4-bit quantization is a small but real loss of model quality in exchange for a large memory saving. The loss tends to be larger for small models, for tasks that require long reasoning chains or precise arithmetic, and for languages that were underrepresented in the calibration data.

SymptomLikely causeRemedy
Perplexity rises sharplyPlain RTN with large groupsUse GPTQ or AWQ with group size 128 or smaller
Garbled output on one taskCalibration data unlike the taskCalibrate with representative samples
No speedup on deviceMissing 4-bit kernels or delegateUse a runtime with native INT4 support
Quality drop in small modelsToo little redundancy to absorb errorKeep sensitive layers at 8 bits
Typical problems and remedies

A frequent misconception is that 4-bit weights always make inference faster. Speed gains depend on kernel support: without optimized dequantization kernels or a suitable hardware delegate, the device may unpack weights inefficiently. Benchmarks should also measure task accuracy, not only perplexity, because perplexity can hide failures on specific skills.

Stripped to the essentials

An AI model is a huge list of numbers. Normally each number is stored with high precision, which takes considerable memory. 4-bit quantization rounds every number to one of only 16 steps, like replacing a detailed ruler with one that has just a few marks.

The model becomes about four times smaller, so it can fit on a phone or laptop. Clever rounding methods keep the answers almost as good as before. The cost is a small drop in quality, which is usually acceptable when the alternative is not running the model on the device at all.

Where it is applied

A legal services team wants a private assistant that summarizes contracts on staff laptops, without sending documents to a cloud service. The chosen open model has 8 billion parameters, which needs about 16 GB in FP16, more than the shared memory available on their 16 GB machines once the operating system and applications are running.

The engineers apply AWQ with group size 128, calibrating on a few hundred anonymized contract paragraphs. The weights shrink to about 4.5 GB, and the model runs in a local inference runtime at roughly 20 to 30 tokens per second. On an internal test set of summaries, reviewers rate the 4-bit output nearly indistinguishable from the FP16 version, and the documents never leave the device.