Data Distillation for Text Classification

Understand

Deep learning techniques have achieved great success in many fields, while at the same time deep learning models are getting more complex and expensive to compute.

• It severely hinders the wide applications of these models.
• In order to alleviate this problem, model distillation emerges as an effective means to compress a large model into a smaller one without a significant drop in accuracy.
• In this paper, we study a related but orthogonal issue, data distillation, which aims to distill the knowledge from a large training dataset down to a smaller and synthetic one.