MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesis

Understand

Generative modeling and representation learning are two key tasks in computer vision.

• However, these models are typically trained independently, which ignores the potential for each task to help the other, and leads to training and model maintenance overheads.
• In this work, we propose MAsked Generative Encoder (MAGE), the first framework to unify SOTA image generation and self-supervised representation learning.
• Our key insight is that using variable masking ratios in masked image modeling pre-training can allow generative training (very high masking ratio) and representation learning (lower masking ratio) under the same training framework.