AdaMAE: Adaptive Masking for Efficient Spatiotemporal Learning with Masked Autoencoders

Understand

Masked Autoencoders (MAEs) learn generalizable representations for image, text, audio, video, etc., by reconstructing masked input data from tokens of the visible data.

• Current MAE approaches for videos rely on random patch, tube, or frame-based masking strategies to select these tokens.
• This paper proposes AdaMAE, an adaptive masking strategy for MAEs that is end-to-end trainable.
• Our adaptive masking strategy samples visible tokens based on the semantic context using an auxiliary sampling network.