Distillation, also called knowledge distillation, is a training method in which a smaller student model learns to imitate a larger teacher model, using the teacher’s outputs as training targets instead of, or in addition to, the original labels. In AI training, it is the standard way to transfer most of a large model’s capability into a model that is cheaper and faster to run.
The idea rests on a simple observation: a trained teacher’s predictions contain more information than hard labels. A probability distribution that says an image is mostly a husky, somewhat a wolf and barely a cat teaches the student how classes relate, which is why a distilled student often outperforms the same small model trained on labels alone.
How distillation transfers knowledge
Distillation works by training the student to match the teacher’s output distribution on a set of inputs, called the transfer set. In classic logit distillation, both models produce scores over the same classes or tokens, and the student minimizes the Kullback-Leibler divergence between its distribution and the teacher’s.
A temperature parameter softens both distributions before comparison. Dividing the logits by a temperature above 1 spreads probability onto less likely classes, exposing the teacher’s so-called dark knowledge about similarities between outputs. The distillation loss is usually combined with an ordinary cross-entropy loss on true labels, weighted by a mixing coefficient.
For large language models, distillation often takes a simpler form: the teacher generates responses to many prompts, and the student is fine-tuned on those responses as if they were human-written data. This sequence-level distillation needs only the teacher’s text, not its internal probabilities, which is why it works even when the teacher is accessible only through an API.
Main forms of distillation
Distillation methods differ in what signal the student imitates and in how much access to the teacher they require. The table summarizes the main families.
| Form | What the student matches | Teacher access needed |
|---|---|---|
| Logit distillation | Softened output probabilities | Full output logits |
| Feature distillation | Intermediate activations or hidden states | Internal layers |
| Attention distillation | Attention maps between tokens | Internal layers |
| Sequence-level distillation | Generated text or labels | Outputs only |
| On-policy distillation | Teacher scores on student-generated text | Logits on new samples |
| Self-distillation | Its own earlier or deeper version | Same architecture |
On-policy distillation addresses a known weakness of training only on teacher outputs: the student never sees its own mistakes. When the teacher grades sequences the student actually produces, the student learns to recover from its own errors, which tends to help on long reasoning tasks.
Key parameters and their typical values
A handful of hyperparameters decide whether distillation succeeds. Their best values depend on the task and the capacity gap between teacher and student, so the ranges below are starting points rather than fixed rules.
| Parameter | Common range | Effect |
|---|---|---|
| Temperature | 1-10, often 2-4 | Higher values expose more dark knowledge |
| Distillation loss weight | 0.5-0.9 | Balance between teacher signal and true labels |
| Student size | About 2x to 10x smaller than teacher | Larger gaps lose more capability |
| Transfer set size | Thousands to billions of tokens | Coverage of the target behavior |
| Learning rate | Similar to fine-tuning of the student | Stability of imitation |
When logits are softened by a temperature, the soft loss is customarily multiplied by the square of the temperature so that its gradients stay comparable in size to the hard-label loss. A very large capacity gap can make the teacher too hard to imitate; an intermediate teacher assistant model is sometimes used to bridge it.
Distillation compared with other compression and training methods
Distillation is one of several ways to make models cheaper, and it is often combined with them. Unlike quantization or pruning, it produces a new, separately trained model rather than modifying the existing one.
| Technique | What changes | Typical gain | Main cost |
|---|---|---|---|
| Distillation | New smaller architecture trained | Large reductions in size and latency | Training compute and data |
| Quantization | Numeric precision of weights | About 2x to 4x less memory | Small accuracy loss |
| Pruning | Weights or structures removed | Variable, hardware dependent | Retraining to recover accuracy |
| Fine-tuning | Weights adapted to a task | Better task quality | Labeled or curated data |
| Transfer learning | Pretrained features reused | Less task data needed | Domain mismatch risk |
A common production pipeline distills first, then quantizes: the student captures the teacher’s behavior in fewer parameters, and quantization then shrinks the memory footprint further. Distillation can also be applied after RLHF or DPO, carrying a teacher’s aligned behavior into a smaller assistant.
Limitations and common mistakes
Distillation rarely produces a student that fully matches its teacher, and the losses are uneven across tasks. Narrow tasks distill well, while broad knowledge and complex multi-step reasoning degrade first as the student shrinks.
- Copying teacher errors: the student inherits the teacher’s biases and hallucinations along with its strengths.
- Narrow transfer sets: a student distilled on one domain loses general ability outside it.
- Overconfident imitation: training only on hard teacher outputs discards the uncertainty information.
- Evaluation on teacher-like data only: scores look high on the transfer distribution and fall elsewhere.
- Licensing conflicts: many model terms of use restrict using outputs to train competing models.
The most reliable safeguard is to evaluate the student on held-out benchmarks that reflect real usage, including cases the teacher handles poorly, and to filter teacher outputs for correctness before training on them.
A plain-language summary
Distillation is like an experienced professor preparing a compact study guide for a student. The professor knows far more than fits in the guide, but by showing not just the right answers but also which wrong answers were close and why, the guide teaches more than a plain answer key would.
The student ends up much quicker and cheaper to consult than the professor, while handling most everyday questions almost as well. The trade-off is that some depth is lost, especially on the hardest and rarest questions.
An example from practice
A customer support team runs a large language model that classifies incoming tickets into about forty categories and drafts first replies. The large model performs well, but it is too slow and costly for tens of thousands of tickets a day.
The team uses the large model as a teacher to label and answer a few hundred thousand historical tickets, filters out answers that agents had corrected, and fine-tunes a model roughly ten times smaller on the result, keeping the teacher’s class probabilities for the categorization head. After evaluation on a held-out month of tickets, the student reaches classification accuracy within a few points of the teacher at a fraction of the latency, and the team routes only low-confidence tickets back to the large model.