Towards Understanding Why Mask-Reconstruction Pretraining Helps in Downstream Tasks

Understand

For unsupervised pretraining, mask-reconstruction pretraining (MRP) approaches, e.g.

• MAE and data2vec, randomly mask input patches and then reconstruct the pixels or semantic features of these masked patches via an auto-encoder.
• Then for a downstream task, supervised fine-tuning the pretrained encoder remarkably surpasses the conventional ``supervised learning'' (SL) trained from scratch.
• However, it is still unclear 1) how MRP performs semantic feature learning in the pretraining phase and 2) why it helps in downstream tasks.