GPTQ is a one-shot post-training quantization method that compresses the weights of large transformer models to 3 or 4 bits per weight while keeping accuracy close to the original 16-bit model. In Edge AI, it is one of the standard ways to make a large language model small enough to run on a single consumer GPU, a laptop or a workstation instead of a data center cluster.
The method was introduced in 2022 by Elias Frantar, Saleh Ashkboos, Torsten Hoefler and Dan Alistarh. Its name refers to quantizing generative pre-trained transformers, and its key idea is to correct quantization error as it happens, using approximate second-order information collected from a small calibration dataset.
How GPTQ compensates quantization error
GPTQ quantizes a model layer by layer, treating each linear layer as a separate reconstruction problem. The goal is to find low-bit weights whose outputs on real inputs stay as close as possible to the outputs of the original full-precision weights, measured as a squared output error.
The method builds on Optimal Brain Quantization, which descends from the classic Optimal Brain Surgeon pruning work. For every layer it computes the Hessian of the layer-wise loss, which equals twice the product of the calibration inputs with their transpose. This matrix describes how sensitive the output is to changes in each input dimension.
GPTQ then processes the weight matrix one column at a time. After rounding a column to the quantization grid, it spreads the resulting error over the remaining unquantized columns, adjusting them so that the layer output is preserved. Later columns therefore absorb the mistakes of earlier ones, which is why the result is far better than plain rounding.
Three engineering choices made the approach practical for models with hundreds of billions of parameters. They are summarized below.
| Technique | What it does | Effect |
|---|---|---|
| Fixed column order | Quantizes all rows in the same column order | Inverse Hessian is shared across rows |
| Lazy batch updates | Applies updates in blocks of about 128 columns | Better GPU memory bandwidth use |
| Cholesky reformulation | Precomputes needed rows of the inverse Hessian | Numerical stability on very large layers |
| Dampening | Adds a small value, often 1% of the mean diagonal, to the Hessian | Prevents unstable inversion |
Key parameters of a GPTQ run
A GPTQ checkpoint is defined by a handful of settings that determine file size, speed and accuracy. Choosing them well matters more than the choice of toolkit, because every popular implementation exposes the same core options.
| Parameter | Typical value | What it affects |
|---|---|---|
| Bit width | 4 bits, sometimes 3 or 8 | Memory footprint and accuracy |
| Group size | 128 weights per scale | Accuracy versus metadata overhead |
| Activation order | On or off | Accuracy on outlier-heavy layers |
| Symmetric grid | Symmetric or asymmetric | Zero-point storage and kernel support |
| Calibration set | About 128 sequences of 2,048 tokens | Quality of the Hessian estimate |
| Damping percent | 0.01 to 0.1 | Stability of the error correction |
Group size has a large effect at low bit widths. With a group size of 128, each block of 128 weights receives its own scale and zero point, which adds roughly 0.15 to 0.25 bits per weight of overhead but recovers much of the accuracy lost at 3 or 4 bits. Activation order, often called act-order or desc_act, quantizes the columns with the largest Hessian diagonal first; it usually improves accuracy, although older inference kernels ran it more slowly.
The calibration data should resemble the text the model will see in use. Generic web text is the usual default, but a model that will answer code questions benefits from code samples, and a multilingual model needs multilingual samples.
GPTQ compared with other weight quantization methods
GPTQ is a weight-only quantization method: weights are stored in low precision, while activations stay in 16-bit floating point and weights are dequantized on the fly inside the matrix multiplication. It sits among several related techniques in the same Topic.
| Method | Core idea | Main trade-off |
|---|---|---|
| Round-to-nearest | Rounds each weight independently | Fast but loses accuracy below 8 bits |
| GPTQ | Second-order error compensation per layer | Needs calibration data and a GPU |
| AWQ | Scales salient channels based on activations | Simpler, often similar 4-bit accuracy |
| SmoothQuant | Moves outliers from activations into weights | Targets 8-bit weights and activations |
| Quantization-aware training | Simulates quantization during training | Best accuracy, highest compute cost |
In practice, GPTQ and AWQ produce comparable perplexity at 4 bits on most open models, and the better choice often depends on available inference kernels rather than on accuracy. At 3 bits, GPTQ with small group sizes tends to hold up better than simple rounding, but both methods show measurable degradation at that level.
Deploying GPTQ models on local hardware
GPTQ makes local LLM inference possible on hardware with limited memory. A 7-billion-parameter model needs about 14 GB in 16-bit precision but roughly 4 GB at 4 bits, which fits on a mid-range consumer GPU together with the key-value cache.
The speed benefit comes from memory bandwidth, not from cheaper arithmetic. Token generation at small batch sizes is limited by how fast weights move from memory, so reading a quarter of the bytes can deliver two to four times faster decoding when an optimized kernel such as Marlin or ExLlama is available.
- Prefer 4-bit with group size 128 as the default balance of size and quality.
- Use 8-bit GPTQ when accuracy is critical and memory is not the main constraint.
- Check that the target runtime supports the chosen bit width and act-order setting.
- Validate the result with perplexity and task benchmarks, not with a few chat prompts.
Limitations and common mistakes with GPTQ
GPTQ is effective but not free of pitfalls. Most problems come from mismatched calibration data, unsupported kernels or unrealistic expectations about very low bit widths.
| Symptom | Likely cause | Fix |
|---|---|---|
| Accuracy drop on code tasks | Calibration used only prose | Add domain samples to calibration |
| Slow generation | Generic dequantization kernel | Use a runtime with optimized kernels |
| Garbled output at 2 bits | Bit width below practical limit | Return to 3 or 4 bits |
| Quantization crash | Ill-conditioned Hessian | Increase the damping percent |
GPTQ also does not reduce activation memory or the key-value cache, which can dominate memory use with long contexts. It is a GPU-oriented format, so CPU-centric runtimes usually rely on their own quantization formats instead. Finally, the method needs a GPU to quantize large models, although a single modern accelerator is enough for most open models.
The short version
Large AI language models store billions of numbers, and each number normally takes 16 bits. GPTQ shrinks each number to about 4 bits, so the whole model becomes roughly four times smaller and can run on an ordinary graphics card.
Simply rounding every number would damage the model. GPTQ rounds the numbers one group at a time and nudges the remaining numbers to cancel the error, a bit like a bookkeeper who corrects later entries so the final total still balances. The result is a compact model that answers almost as well as the original.
Where it makes a difference
A small legal services firm wants an internal assistant that summarizes contracts without sending documents to an external cloud service. Its only machine is a workstation with a 24 GB consumer GPU, which cannot hold a 32-billion-parameter model in 16-bit precision, since that would require about 64 GB.
The engineer quantizes the model with GPTQ at 4 bits and group size 128, using a few hundred passages from the firm’s own anonymized contracts as calibration data. The quantized weights occupy about 18 GB, leaving room for a long context window. On the firm’s internal summary test set, the quantized model scores within one point of the full-precision version, and all documents stay on premises.