How can a small AI model learn to match a giant one?

The answer

Knowledge distillation is a technique where a large, powerful 'teacher' model transfers its learned capabilities to a smaller, faster 'student' model. Rather than simply copying the teacher's final predictions, the student learns from 'soft labels' - the full probability distributions the teacher assigns to all possible outputs, capturing nuanced relationships like how a '7' slightly resembles a '1.' This richer signal allows the student to absorb the teacher's implicit knowledge. The technique, first formalized by Geoffrey Hinton in 2015, has become essential in the LLM era: DeepSeek distilled its 671B-parameter R1 model into versions as small as 1.5B parameters that still outperform many larger models on reasoning benchmarks. Typically, distillation achieves 5-10x model compression while retaining 90-95% of original accuracy, enabling deployment on mobile devices and edge hardware.