Condensed Data Expansion Using Model Inversion for Knowledge Distillation

Understand

Condensed datasets offer a compact representation of larger datasets, but training models directly on them or using them to enhance model performance through knowledge distillation (KD) can result in suboptimal outcomes due to limited information.

• To address this, we propose a method that expands condensed datasets using model inversion, a technique for generating synthetic data based on the impressions of a pre-trained model on its training data.
• This approach is particularly well-suited for KD scenarios, as the teacher model is already pre-trained and retains knowledge of the original training data.
• By creating synthetic data that complements the condensed samples, we enrich the training set and better approximate the underlying data distribution, leading to improvements in student model accuracy during knowledge distillation.