Model distillation

Model distillation lets a small AI model learn from a large one. The large model, the teacher, answers many example questions, and the small model, the student, is trained to give the same answers. The student is cheaper and faster to run, and usually somewhat weaker. A customer service desk could distil a large model into a small one that only sorts incoming emails.

How does it work?

The idea goes back to Bucilă, Caruana and Niculescu-Mizil (Cornell, 2006), who compressed large ensembles of models into one small, fast model. Hinton, Vinyals and Dean (Google, 2015) named it distillation. A classification model gives every possible answer a probability, and the wrong answers carry information too. In the paper's example, an image of a BMW has a very small chance of being mistaken for a garbage truck, and that mistake is still many times more likely than mistaking it for a carrot.

Distillation trains the student on these probabilities, called soft targets, often alongside the correct labels. During training both models divide their scores by a temperature above 1. That flattens the probabilities, so the small ones count too; afterwards the student runs at temperature 1. On handwritten digits (MNIST), a large network made 67 errors on the test set and a small network trained the normal way 146. The same small network, trained to match the large one's soft targets at temperature 20, made 74.

How is it used for language models?

DistilBERT (Hugging Face, 2019) has 66 million parameters instead of BERT's 110 million. It kept 97% of BERT's score on GLUE, a language-understanding benchmark, and ran 60% faster on a CPU. Google DeepMind trained the 2B and 9B versions of Gemma 2 (2024) on the probabilities the teacher gave every next token. In the team's test, a 2B model distilled from a 7B model averaged 67.7 on three benchmarks, against 60.3 when trained from scratch.

Without access to the teacher's probabilities, for example with a model behind an API, the student is fine-tuned on the teacher's answers. Kim and Rush (2016) called this sequence-level knowledge distillation. DeepSeek used it for DeepSeek-R1 (January 2025): it fine-tuned smaller Qwen and Llama models on about 800,000 examples curated with R1. The distilled 32B model scored 72.6% on the AIME 2024 maths test, against 79.8% for R1 itself and 47.0% for the same base model trained with reinforcement learning alone. In October 2024 OpenAI added a distillation workflow to its API: you store the answers of a large model such as GPT-4o and use them to fine-tune a smaller one such as GPT-4o mini.

Distillation builds a new, smaller model. Quantization saves in a different way: it keeps the same model and stores its numbers with less precision.

Where does it go wrong?

• The student can copy the style without the knowledge. Gudibande et al. (2023) fine-tuned models of 1.5 to 13 billion parameters on ChatGPT output. Crowd workers rated the answers as competitive with ChatGPT, while targeted tests showed that on tasks the imitation data covered poorly, the imitations closed little to none of the gap.
• The teacher's owner can forbid it. Anthropic's commercial terms bar customers from using its services to train competing AI models. In February 2026 Anthropic reported that DeepSeek, Moonshot and MiniMax had generated more than 16 million exchanges with Claude through about 24,000 fraudulent accounts to improve their own models. The same post calls distillation a widely used, legitimate method that AI labs routinely apply to their own models.

Example

Suppose the customer service desk of a webshop has a large model sort incoming emails into ten categories. It saves a few thousand of those emails with their labels, fine-tunes a small model on them and compares both models on emails neither has seen. If the small model comes close enough, it takes over the routine sorting and doubtful cases still go to the large model.