Model compression is the set of techniques that make a trained neural network smaller and cheaper to run while preserving as much of its accuracy as possible. In AI infrastructure it is a central lever for inference cost, latency and hardware fit, because a compressed model needs less memory, less bandwidth and fewer arithmetic operations per request.
The main families are quantization, pruning, knowledge distillation and low-rank factorization. They are often combined, for example a distilled model that is then quantized to 4-bit weights so it fits on a single GPU or runs as on-device AI on a phone.
Why compressing a model reduces cost
Large models are usually limited by memory capacity and memory bandwidth rather than raw compute. A model with 70 billion parameters stored in 16-bit precision needs about 140 GB for weights alone, before the KV cache and activations. During token-by-token generation, every weight is read from memory for each step, so halving the bytes per weight roughly halves the time spent moving data.
Compression attacks this bottleneck in different ways: it reduces bytes per parameter (quantization), removes parameters (pruning), trains a smaller model (distillation) or restructures weight matrices (low-rank factorization). The gain is real only when the serving hardware and kernels support the compressed format efficiently.
Smaller weights also free memory for other uses. On a fixed GPU, the space saved by compression can hold a larger KV cache, which allows longer contexts or more concurrent requests in a batch. For many serving teams this higher throughput per GPU is worth more than the reduction in single-request latency, because it directly lowers the cost of each generated token.
Main model compression techniques
The table below compares the four principal techniques by mechanism, typical reduction and cost of application.
| Technique | Mechanism | Typical size reduction | Retraining needed |
|---|---|---|---|
| Quantization | Lower numeric precision of weights or activations | 2x to 4x from 16-bit | Optional |
| Pruning | Remove weights, neurons, heads or layers | Variable, often 1.5x to 2x usable | Usually fine-tuning |
| Knowledge distillation | Train a small student to mimic a teacher | Often 2x to 10x fewer parameters | Yes, full training |
| Low-rank factorization | Replace a matrix with two thin matrices | Moderate, layer dependent | Usually fine-tuning |
Quantization is the most widely used method for large language models. Post-training quantization (PTQ) converts weights after training using a small calibration set; methods such as GPTQ and AWQ keep 4-bit weight quality close to the original. Quantization-aware training (QAT) simulates low precision during training and holds up better at very low bit widths, at higher cost.
Pruning is either unstructured, zeroing individual weights, or structured, removing whole channels, attention heads or layers. Unstructured sparsity reaches high ratios on paper but speeds up inference only with sparse-aware kernels; structured pruning delivers speedups on ordinary hardware but costs more accuracy per removed parameter.
Precision formats and their trade-offs
Quantization choices are usually expressed as numeric formats. Lower precision saves memory but increases quantization error, which shows up first on reasoning-heavy and long-context tasks.
| Format | Bytes per weight | Typical quality impact | Hardware support |
|---|---|---|---|
| FP16 or BF16 | 2 | Reference quality | Universal on modern GPUs |
| FP8 | 1 | Small loss with calibration | Recent data center GPUs |
| INT8 | 1 | Small loss for most models | Broad GPU and CPU support |
| INT4 or FP4 | 0.5 | Noticeable on hard tasks | Growing kernel support |
| 2-3 bit | Below 0.4 | Significant without QAT | Experimental, specialized |
Model compression compared with related serving optimizations
Compression changes the model artifact itself, while many neighboring techniques in AI infrastructure change how an unchanged model is served. They are complementary, and production stacks typically combine several of them.
| Technique | What changes | Main benefit |
|---|---|---|
| Model compression | Weights, precision or architecture | Less memory and compute per token |
| KV cache quantization | Stored attention keys and values | Longer contexts, more concurrent users |
| Speculative decoding | Decoding procedure | Lower latency, same output distribution |
| Prompt caching | Reuse of computed prefixes | Lower cost for repeated prompts |
| Model routing | Which model answers a request | Cheap model for easy requests |
Choosing a compression strategy
Start from the deployment constraint: the memory budget of the target device, the latency target or the cost per million tokens. Then pick the least invasive technique that meets it, because each additional step adds evaluation burden and risk.
| Constraint | Recommended first step | Reason |
|---|---|---|
| Model barely exceeds GPU memory | 8-bit weight quantization | Minimal quality loss, no retraining |
| Running on a laptop or phone | 4-bit quantization of a small model | Fits limited memory and bandwidth |
| Strict latency, narrow task | Distill into a task-specific student | Large reduction with targeted quality |
| Throughput-bound serving | FP8 weights and activations | Faster matrix math on supported GPUs |
Always evaluate the compressed model on the real task distribution, not only on perplexity. Perplexity can barely move while accuracy on code generation, multilingual text or long documents drops by several points.
Risks and common mistakes
- Assuming smaller means faster: unsupported formats can be dequantized on the fly and run slower than the original.
- Calibrating on the wrong data: PTQ calibrated on generic text can hurt domain-specific accuracy.
- Ignoring outlier channels: a few large activation values dominate quantization error unless handled explicitly.
- Compounding losses: pruning, distillation and quantization together can erode quality more than each step suggests.
- Skipping safety evaluation: compression can change refusal and safety behavior, so re-run those tests.
A compressed model should be treated as a new model version in the model registry, with its own evaluation record, rather than as a transparent drop-in replacement.
Put simply
Model compression is like packing for a trip with a smaller suitcase. You fold clothes more tightly, leave out items you rarely use and maybe bring one versatile jacket instead of three. The trip still works, and the bag is far easier to carry.
For AI models, compression stores numbers with fewer digits, removes parts that contribute little or teaches a smaller model to copy a larger one. The result runs on cheaper hardware and answers faster, as long as someone checks that it still answers well.
In practice
A company runs an internal assistant on an open-weight model with about 70 billion parameters. In 16-bit precision it needs two high-memory GPUs per replica, which limits how many replicas the team can afford during peak hours.
The team applies 4-bit weight quantization with a calibration set drawn from real internal queries and evaluates the result on 500 representative tasks. Quality drops by about one point on their internal score, mostly on long summaries, which they accept. Each replica now fits on a single GPU, so the same hardware serves roughly twice as many replicas and queueing delays at peak time largely disappear.