Rethinking Patch Dependence for Masked Autoencoders

Understand

In this work, we examine the impact of inter-patch dependencies in the decoder of masked autoencoders (MAE) on representation learning.

• We decompose the decoding mechanism for masked reconstruction into self-attention between mask tokens and cross-attention between masked and visible tokens.
• Our findings reveal that MAE reconstructs coherent images from visible patches not through interactions between patches in the decoder but by learning a global representation within the encoder.
• This discovery leads us to propose a simple visual pretraining framework: cross-attention masked autoencoders (CrossMAE).