Hybrid Distillation: Connecting Masked Autoencoders with Contrastive Learners

Understand

Representation learning has been evolving from traditional supervised training to Contrastive Learning (CL) and Masked Image Modeling (MIM).

• Previous works have demonstrated their pros and cons in specific scenarios, i.e., CL and supervised pre-training excel at capturing longer-range global patterns and enabling better feature discrimination, while MIM can introduce more local and diverse attention across all transformer layers.
• In this paper, we explore how to obtain a model that combines their strengths.
• We start by examining previous feature distillation and mask feature reconstruction methods and identify their limitations.