Dark knowledge
Entropy of the soft targets
How much the teacher hedges at each temperature. The dashed line is a uniform guess, log k.
Students over epochs (held-out test set)
What is knowledge distillation?
A big model (the teacher) is trained first. A smaller student is then trained to imitate the teacher's full output distribution instead of, or as well as, the one-hot labels. Buciluǎ et al. (2006) compressed large ensembles this way by labelling a large synthetic transfer set with the ensemble. Ba & Caruana (2014) trained shallow nets to regress a deep net's logits. Hinton et al. (2015) gave the now-standard recipe: soften both distributions with a temperature T and mix in the ordinary label loss:
L = α · T² · KL( softmax(v/T) ‖ softmax(z/T) ) + (1 − α) · CE( y, softmax(z) )
where v are the teacher's logits and z the student's. Here both students have the same size and the same initial weights; the only difference is the loss. The KL term is averaged over every training point (labelled or not) and the CE term over the labelled ones.
Why do soft targets carry information?
A one-hot label says only which class. The teacher's probabilities also say how sure it is and which wrong classes are plausible: a point near a boundary gets something like 0.6 / 0.4, a point deep inside a class gets 0.99 / 0.01. Hinton et al. call these small probabilities dark knowledge. They describe the shape of the teacher's function around each point, so one soft target carries more bits than one label. And they can be computed on unlabelled points, so the transfer set can be far larger than the labelled set. That is why distillation shines in the "few labels" preset.
At T = 1 a confident teacher's targets are nearly one-hot, so the small probabilities are invisible. Raising T divides the logits before the softmax, which lifts those small probabilities into view. Too high, and every class looks equally likely.
Why multiply by T²?
The gradient of the soft term with respect to a student logit is
∂/∂zi [ T² · KL ] = T · ( qi − pi ), q = softmax(z/T), p = softmax(v/T).
For large T, qi − pi ≈ (zi − vi) / (k·T) (for zero-mean logits), so without the T² factor the soft gradient would shrink like 1/T² and changing T would silently change the balance set by α. With it, the gradient stays roughly (zi − vi)/k: in the high-temperature limit distillation becomes logit matching, the objective of Ba & Caruana.
References
- G. Hinton, O. Vinyals, J. Dean. Distilling the Knowledge in a Neural Network. arXiv:1503.02531, 2015.
- C. Buciluǎ, R. Caruana, A. Niculescu-Mizil. Model Compression. KDD, 2006.
- J. Ba, R. Caruana. Do Deep Nets Really Need to be Deep? NeurIPS, 2014.