Few Shot Network Compression via Cross Distillation

Understand

Model compression has been widely adopted to obtain light-weighted deep neural networks.

• Most prevalent methods, however, require fine-tuning with sufficient training data to ensure accuracy, which could be challenged by privacy and security issues.
• As a compromise between privacy and performance, in this paper we investigate few shot network compression: given few samples per class, how can we effectively compress the network with negligible performance drop? The core challenge of few shot network compression lies in high estimation errors from the original network during inference, since the compressed network can easily over-fits on the few training instances.
• The estimation errors could propagate and accumulate layer-wisely and finally deteriorate the network output.