GPTQ

On this page 7

GPTQ is a one-shot post-training quantization method that compresses the weights of large transformer models to 3 or 4 bits per weight while keeping accuracy close to the original 16-bit model. In Edge AI, it is one of the standard ways to make a large language model small enough to run on a single consumer GPU, a laptop or a workstation instead of a data center cluster.

The method was introduced in 2022 by Elias Frantar, Saleh Ashkboos, Torsten Hoefler and Dan Alistarh. Its name refers to quantizing generative pre-trained transformers, and its key idea is to correct quantization error as it happens, using approximate second-order information collected from a small calibration dataset.

How GPTQ compensates quantization error

GPTQ quantizes a model layer by layer, treating each linear layer as a separate reconstruction problem. The goal is to find low-bit weights whose outputs on real inputs stay as close as possible to the outputs of the original full-precision weights, measured as a squared output error.

The method builds on Optimal Brain Quantization, which descends from the classic Optimal Brain Surgeon pruning work. For every layer it computes the Hessian of the layer-wise loss, which equals twice the product of the calibration inputs with their transpose. This matrix describes how sensitive the output is to changes in each input dimension.

GPTQ then processes the weight matrix one column at a time. After rounding a column to the quantization grid, it spreads the resulting error over the remaining unquantized columns, adjusting them so that the layer output is preserved. Later columns therefore absorb the mistakes of earlier ones, which is why the result is far better than plain rounding.

Three engineering choices made the approach practical for models with hundreds of billions of parameters. They are summarized below.

TechniqueWhat it doesEffect
Fixed column orderQuantizes all rows in the same column orderInverse Hessian is shared across rows
Lazy batch updatesApplies updates in blocks of about 128 columnsBetter GPU memory bandwidth use
Cholesky reformulationPrecomputes needed rows of the inverse HessianNumerical stability on very large layers
DampeningAdds a small value, often 1% of the mean diagonal, to the HessianPrevents unstable inversion
Engineering steps that make GPTQ scale

Key parameters of a GPTQ run

A GPTQ checkpoint is defined by a handful of settings that determine file size, speed and accuracy. Choosing them well matters more than the choice of toolkit, because every popular implementation exposes the same core options.

ParameterTypical valueWhat it affects
Bit width4 bits, sometimes 3 or 8Memory footprint and accuracy
Group size128 weights per scaleAccuracy versus metadata overhead
Activation orderOn or offAccuracy on outlier-heavy layers
Symmetric gridSymmetric or asymmetricZero-point storage and kernel support
Calibration setAbout 128 sequences of 2,048 tokensQuality of the Hessian estimate
Damping percent0.01 to 0.1Stability of the error correction
Main GPTQ settings and typical values

Group size has a large effect at low bit widths. With a group size of 128, each block of 128 weights receives its own scale and zero point, which adds roughly 0.15 to 0.25 bits per weight of overhead but recovers much of the accuracy lost at 3 or 4 bits. Activation order, often called act-order or desc_act, quantizes the columns with the largest Hessian diagonal first; it usually improves accuracy, although older inference kernels ran it more slowly.

The calibration data should resemble the text the model will see in use. Generic web text is the usual default, but a model that will answer code questions benefits from code samples, and a multilingual model needs multilingual samples.

GPTQ compared with other weight quantization methods

GPTQ is a weight-only quantization method: weights are stored in low precision, while activations stay in 16-bit floating point and weights are dequantized on the fly inside the matrix multiplication. It sits among several related techniques in the same Topic.

MethodCore ideaMain trade-off
Round-to-nearestRounds each weight independentlyFast but loses accuracy below 8 bits
GPTQSecond-order error compensation per layerNeeds calibration data and a GPU
AWQScales salient channels based on activationsSimpler, often similar 4-bit accuracy
SmoothQuantMoves outliers from activations into weightsTargets 8-bit weights and activations
Quantization-aware trainingSimulates quantization during trainingBest accuracy, highest compute cost
How GPTQ relates to neighboring techniques

In practice, GPTQ and AWQ produce comparable perplexity at 4 bits on most open models, and the better choice often depends on available inference kernels rather than on accuracy. At 3 bits, GPTQ with small group sizes tends to hold up better than simple rounding, but both methods show measurable degradation at that level.

Deploying GPTQ models on local hardware

GPTQ makes local LLM inference possible on hardware with limited memory. A 7-billion-parameter model needs about 14 GB in 16-bit precision but roughly 4 GB at 4 bits, which fits on a mid-range consumer GPU together with the key-value cache.

The speed benefit comes from memory bandwidth, not from cheaper arithmetic. Token generation at small batch sizes is limited by how fast weights move from memory, so reading a quarter of the bytes can deliver two to four times faster decoding when an optimized kernel such as Marlin or ExLlama is available.

  • Prefer 4-bit with group size 128 as the default balance of size and quality.
  • Use 8-bit GPTQ when accuracy is critical and memory is not the main constraint.
  • Check that the target runtime supports the chosen bit width and act-order setting.
  • Validate the result with perplexity and task benchmarks, not with a few chat prompts.

Limitations and common mistakes with GPTQ

GPTQ is effective but not free of pitfalls. Most problems come from mismatched calibration data, unsupported kernels or unrealistic expectations about very low bit widths.

SymptomLikely causeFix
Accuracy drop on code tasksCalibration used only proseAdd domain samples to calibration
Slow generationGeneric dequantization kernelUse a runtime with optimized kernels
Garbled output at 2 bitsBit width below practical limitReturn to 3 or 4 bits
Quantization crashIll-conditioned HessianIncrease the damping percent
Typical GPTQ problems and their fixes

GPTQ also does not reduce activation memory or the key-value cache, which can dominate memory use with long contexts. It is a GPU-oriented format, so CPU-centric runtimes usually rely on their own quantization formats instead. Finally, the method needs a GPU to quantize large models, although a single modern accelerator is enough for most open models.

The short version

Large AI language models store billions of numbers, and each number normally takes 16 bits. GPTQ shrinks each number to about 4 bits, so the whole model becomes roughly four times smaller and can run on an ordinary graphics card.

Simply rounding every number would damage the model. GPTQ rounds the numbers one group at a time and nudges the remaining numbers to cancel the error, a bit like a bookkeeper who corrects later entries so the final total still balances. The result is a compact model that answers almost as well as the original.

Where it makes a difference

A small legal services firm wants an internal assistant that summarizes contracts without sending documents to an external cloud service. Its only machine is a workstation with a 24 GB consumer GPU, which cannot hold a 32-billion-parameter model in 16-bit precision, since that would require about 64 GB.

The engineer quantizes the model with GPTQ at 4 bits and group size 128, using a few hundred passages from the firm’s own anonymized contracts as calibration data. The quantized weights occupy about 18 GB, leaving room for a long context window. On the firm’s internal summary test set, the quantized model scores within one point of the full-precision version, and all documents stay on premises.