JOURNAL / MODEL TRAINING & EFFICIENCY

What is knowledge distillation?
One model teaches another.

Knowledge distillation trains one model to learn from signals produced by another. It can help build a model that fits a tighter budget, but a smaller student still has to prove it can do the intended work.

Glass laboratory flasks in a dim room, with a central flask sharply lit against softer shapes.TEACHER / STUDENT / SYMPLOKA
BEHAVIOR TRANSFERRED THROUGH TRAINING

The student learns a pattern, not a perfect copy.

Distillation uses a teacher’s outputs or internal signals to train another model. The student develops its own parameters and its own limits.

01 / TEACHER & STUDENT

Train a second model from more than a single answer.

In knowledge distillation, the teacher supplies a learning signal and the student is trained to match selected parts of its behavior. The student is often smaller or cheaper to run, but that is a design goal—not part of a guarantee that the result will fit every task.

A teacher can provide a probability distribution over possible outputs rather than only its top answer. That softer signal can preserve relationships among alternatives that a single hard label leaves out. For example, if a classifier considers two categories plausible, their relative scores give the student more to learn than a label that says only which category won.

Hinton, Vinyals, and Dean’s 2015 paper formalized a method for transferring information from a cumbersome model or ensemble to a smaller model using these “soft targets.” Later methods have taught different signals too; distillation is a family of training approaches rather than one fixed recipe.

Distillation is training from another model’s signals. It does not copy all of the teacher’s weights or automatically reproduce every capability.

02 / WHAT CAN BE TRANSFERRED?

The teaching signal depends on the model and the task.

Output probabilitiesThe student learns to match the teacher’s distribution over classes or next tokens. Relative scores can expose alternatives beyond the top prediction.
Intermediate representationsA training objective can ask the student to match selected hidden states or other internal features from the teacher.
Attention relationshipsFor Transformer models, methods such as MiniLM transfer relations learned inside the teacher’s self-attention layers.
Ground-truth examplesTraining can combine a teacher’s signal with labels, demonstrations, or other task data. The student’s behavior depends on both the data and the objective.

These signals are not interchangeable. The DistilBERT paper, for example, combines language-modeling, distillation, and cosine-distance objectives when training its compact student. MiniLM instead studies how to transfer self-attention relations. The choice changes what the student is asked to preserve.

Distillation also appears in systems outside text classification. The shared idea is to use one model’s outputs or representations to guide another model’s training; details vary by modality and objective.

03 / THE TRAINING LOOP

The teacher can train offline; the student serves the request.

  1. Choose a target job. Define the inputs, required behavior, and operating constraints before selecting a student architecture.
  2. Build a suitable transfer set. Use examples that represent the task and the conditions where the deployed model must work.
  3. Collect teacher signals. Run the teacher on those examples and retain the outputs or representations the chosen method needs.
  4. Train the student. Optimize it against the teacher signal, sometimes alongside verified labels or another training objective.
  5. Evaluate the complete choice. Compare the student with the teacher and a simpler baseline on held-out task examples and target hardware.

In a common setup, the teacher is used during training and only the student is needed at inference time. This can make serving more economical. If the deployed workflow still calls the teacher for each request, measure it as a two-model system; the student alone does not describe its latency, cost, or data path.

04 / DISTILLATION & OTHER COMPRESSION

Similar goals, different mechanisms.

DistillationTrains a student from a teacher’s outputs, features, or other signals. It changes the student through learning.
Fine-tuningContinues training a model on examples or another objective. Fine-tuning and distillation can overlap: a pretrained student can be fine-tuned with a distillation loss.
QuantizationRepresents weights or calculations at lower numerical precision. It can reduce memory use; its speed and quality effects depend on the runtime and hardware.
PruningRemoves or sparsifies selected parameters or structures. A pruned model may need further training to recover task quality.

Teams can combine methods, such as distilling a student and then quantizing it for a device. That makes whole-system measurement essential: a parameter count or compressed file size alone cannot predict memory use, response time, energy, or accepted work. Continue with the quantization guide ↗ and the guide to small language models ↗.

05 / WHAT THE PAPERS SHOW

A result belongs to its method, data, and test.

The DistilBERT authors reported a student with 40% fewer parameters than BERT that retained 97% of its language-understanding capability and ran 60% faster in their evaluated setup. MiniLM reported that its distilled model used 50% of the teacher’s Transformer parameters and computations while retaining over 99% accuracy on SQuAD 2.0 and several GLUE tasks. Those are results for particular models, objectives, benchmarks, and runtimes—not default outcomes for a new deployment.

A student can miss rare cases, lose capabilities not represented in its transfer data, or reproduce a teacher’s mistakes. Test its behavior directly, including unsupported inputs and cases where the correct response is to abstain. A teacher’s strong benchmark score does not transfer automatically.

When comparing candidates, track task quality, edge-case performance, latency, peak memory, power, and cost per accepted result. For a broader selection process, see how to choose a model for real work ↗ and how to read benchmark results ↗.

PRIMARY PAPERSHinton, Vinyals & Dean, Distilling the Knowledge in a Neural Network (2015) ↗Sanh et al., DistilBERT: smaller, faster, cheaper and lighter (2019) ↗Wang et al., MiniLM: Deep Self-Attention Distillation (NeurIPS 2020) ↗