Learning from a Teacher using Unlabeled Data

Understand

Knowledge distillation is a widely used technique for model compression.

• We posit that the teacher model used in a distillation setup, captures relationships between classes, that extend beyond the original dataset.
• We empirically show that a teacher model can transfer this knowledge to a student model even on an {\it out-of-distribution} dataset.
• Using this approach, we show promising results on MNIST, CIFAR-10, and Caltech-256 datasets using unlabeled image data from different sources.