Multi-teacher knowledge distillation as an effective method for compressing ensembles of neural networks

Understand

Deep learning has contributed greatly to many successes in artificial intelligence in recent years.

• Today, it is possible to train models that have thousands of layers and hundreds of billions of parameters.
• Large-scale deep models have achieved great success, but the enormous computational complexity and gigantic storage requirements make it extremely difficult to implement them in real-time applications.
• On the other hand, the size of the dataset is still a real problem in many domains.