Model Distillation — AI Glossary

Model distillation is a technique for creating a smaller, more efficient "student" model that approximates the behavior of a larger "teacher" model. The student is trained not on the original dataset but on the teacher's outputs (including its probability distributions over tokens), allowing it to capture much of the teacher's capability at a fraction of the computational cost. Distillation enables deploying powerful AI capabilities on resource-constrained environments.