Model Distillation
Model distillation is a technique for training a smaller, faster "student" model to replicate the behavior of a larger, more capable "teacher" model, transferring much of its performance at a fraction of the size.
Definition
Model distillation is a technique for training a smaller, faster "student" model to replicate the behavior of a larger, more capable "teacher" model, transferring much of its performance at a fraction of the size.
Overview
Large, highly capable models are often too slow, expensive, or resource-intensive to deploy in production, particularly on edge devices or in applications with strict latency requirements. Model distillation addresses this by training a smaller student model not just on the original labeled training data, but on the outputs — often the full probability distributions rather than just final answers — produced by a larger teacher model, which contain richer information about the teacher's learned patterns than hard labels alone. This process, closely related to transfer learning, allows the student model to approximate much of the teacher's capability while requiring significantly less compute, memory, and latency at inference time. Distillation is one of several model-compression techniques used together in production systems, alongside model quantization, which reduces the numeric precision of a model's weights, and model pruning, which removes unnecessary parameters. Distillation is widely used to create efficient, deployable versions of large foundation models — for example, producing a smaller variant of a large language model that retains much of its capability while running faster and cheaper, making advanced AI more accessible for real-time or resource-constrained applications.
Key Concepts
- Trains a smaller student model to replicate a larger teacher model's behavior
- Uses the teacher's output distributions, not just final labels, as training signal
- Significantly reduces compute, memory, and latency at inference time
- Closely related to transfer learning techniques
- Often combined with quantization and pruning for further compression
- Widely used to create efficient, deployable versions of large foundation models