What is knowledge distillation?

Knowledge distillation is a technique for training a smaller student model to reproduce the behavior of a larger teacher model, transferring much of the teacher's capability into a cheaper, faster model.

What are soft labels in distillation?

Soft labels are the teacher model's full probability distribution over possible outputs, rather than just the single correct answer; they carry richer information about the teacher's reasoning that helps the student learn more effectively.

How is distillation different from fine-tuning?

Fine-tuning typically trains a model on human-labeled ground-truth data, while distillation specifically trains a student model on a stronger teacher model's outputs, which can include reasoning traces or full probability distributions.

Why distill a large language model instead of training a small one from scratch?

A small model trained directly on raw data usually underperforms one trained via distillation, because the teacher's outputs provide a richer, more informative training signal than raw labels alone.

Is knowledge distillation the same as quantization?

No, quantization reduces the numerical precision of an existing model's weights, while distillation trains an entirely separate, typically smaller model to mimic a larger one; the two techniques are often combined.

Which models are examples of distillation in practice?

Several compact open models, including some Nemotron and Phi-family releases, use distillation from larger teacher models as part of their training process to achieve strong performance at a smaller size.

AI Fundamentals

Knowledge Distillation

IntermediateTechnique9.7K learners

Knowledge distillation is a machine learning technique for training a smaller "student" model to reproduce the behavior of a larger, more capable "teacher" model, transferring much of the teacher's performance into a model that is cheaper…

#KnowledgeDistillation #AIFundamentals#Technique#Intermediate#MixtureOfExperts #SpeculativeDecoding #Phi4 #Nemotron #ArtificialIntelligence#Glossary#SkillVeris

Definition

Overview

Large, high-capacity models are often expensive and slow to run in production, while smaller models trained from scratch on the same raw data typically underperform them. Knowledge distillation bridges this gap by training the smaller student model not just on raw labeled data, but on the outputs of the larger teacher model — often its full probability distribution over possible outputs ("soft labels") rather than just the single correct answer ("hard labels"), since the teacher's confidence and relative rankings across all options carry additional information about how it reasons that a single ground-truth label doesn't capture. In the context of large language models, distillation is commonly used to compress a large frontier model into a smaller one that retains much of its capability at a fraction of the inference cost, or to specialize a compact model for a narrower task by training it on the outputs a large general-purpose model produces for that task. This can involve generating synthetic training examples with the teacher model, having the student mimic the teacher's step-by-step reasoning traces, or directly matching internal representations between the two models, depending on the specific distillation method used. Distillation is one of several standard model compression techniques, often used alongside quantization and pruning, and is distinct from simply fine-tuning a small model on human-labeled data because it specifically leverages a stronger teacher's outputs as a richer training signal. It has become especially prominent in the era of frontier LLMs, where distilling a large proprietary or open model into a smaller openly deployable one is a common way to make advanced capabilities more broadly accessible and affordable.

Key Concepts

Trains a smaller student model using outputs from a larger teacher model
Often uses "soft labels" — full output probability distributions — as training signal
Transfers much of a large model's capability into a cheaper, faster model
Can involve mimicking step-by-step reasoning traces from the teacher
Commonly used to compress frontier LLMs into smaller deployable variants
Distinct from standard fine-tuning by leveraging a stronger model's outputs directly
Often combined with quantization and pruning for further model compression
Enables broader, cheaper access to capabilities originally developed in larger models

Use Cases

Compressing a large frontier LLM into a smaller, cheaper-to-serve model

Creating specialized small models for narrow tasks using a general-purpose teacher

Reducing inference latency and cost for production deployment

Generating high-quality synthetic training data from a strong teacher model

Enabling on-device or edge deployment of models with reduced compute budgets

Improving small open-weight models by training them on stronger models' outputs

Frequently Asked Questions

From the Blog

Projects & Case Studies

Project: Build a Data Dashboard with Python and Streamlit

Streamlit turns a Python script into an interactive web app in minutes — no frontend knowledge required. In this project you'll build a live sales dashboard with filters, KPI metrics, and Plotly charts from a CSV dataset, then share it online for free.