2022

How to Understand Masked Autoencoders

Cao, Shuhao, Xu, Peng, Clifton, David A.

Understand

"Masked Autoencoders (MAE) Are Scalable Vision Learners" revolutionizes the self-supervised learning method in that it not only achieves the state-of-the-art for image pre-training, but is also a milestone that bridges the gap between visual and linguistic masked autoencoding (BERT-style) pre-trainings.

However, to our knowledge, to date there are no theoretical perspectives to explain the powerful expressivity of MAE.
In this paper, we, for the first time, propose a unified theoretical framework that provides a mathematical understanding for MAE.
Specifically, we explain the patch-based attention approaches of MAE using an integral kernel under a non-overlapping domain decomposition setting.

Built on

Solution of incorrectly formulated problems and the regularization method
Andrei Nikolajevits Tihonov · 1963
Earlier work this paper cites.
Potential Theory and Function Theory for Irregular Regions
I.U.D. Burago, J.D. Burago, and V.G. Maz’ya · 1969
Earlier work this paper cites.
The theory of tikhonov regularization for fredholm equations
CW Groetsch · 1984
Earlier work this paper cites.
A reliable and fast method for the solution of fredhol integral equations of the first kind based on tikhonov regularization
Jürgen Weese · 1992
Earlier work this paper cites.
Universal approximation bounds for superpositions of a sigmoidal function
Andrew R Barron · 1993
Earlier work this paper cites.
Domain decomposition algorithms
Tony F Chan and Tarek P Mathew · 1994
Earlier work this paper cites.
Lectures on Cauchy’s problem in linear partial differential equations
Jacques Hadamard · 2003
Earlier work this paper cites.
Domain decomposition methods-algorithms and theory
Andrea Toselli and Olof Widlund · 2004
Earlier work this paper cites.
A review of image denoising algorithms, with a new one
Antoni Buades, Bartomeu Coll, and Jean-Michel Morel · 2005
Earlier work this paper cites.
Vector valued reproducing kernel hilbert spaces of integrable functions and mercer theorem
Claudio Carmeli, Ernesto De Vito, and Alessandro Toigo · 2006
Earlier work this paper cites.
Analysis II
C. Gerhardt · 2006
Earlier work this paper cites.
Extracting and composing robust features with denoising autoencoders
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol · 2008
Earlier work this paper cites.
Numerical solution of fredholm integral equations of the second kind
Kendall Atkinson and Weimin Han · 2009
Earlier work this paper cites.
Topology and data
Gunnar Carlsson · 2009
Earlier work this paper cites.
Imagenet: A large-scale hierarchical image database
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei · 2009
Earlier work this paper cites.
Eigenvalues of integral operators defined by smooth positive definite kernels
JC Ferreira and VA2501172 Menegatto · 2009
Earlier work this paper cites.
Bregmanized nonlocal regularization for deconvolution and sparse reconstruction
Xiaoqun Zhang, Martin Burger, Xavier Bresson, and Stanley Osher · 2010
Earlier work this paper cites.
Reproducing kernel Hilbert spaces in probability and statistics
Alain Berlinet and Christine Thomas-Agnan · 2011
Earlier work this paper cites.

Similar

Bounded integral operators on L 2 spaces
Paul Richard Halmos and Viakalathur Shankar Sunder · 2012
Cited alongside, same era.
Linear Integral Equations
Rainer Kress · 2014
Cited alongside, same era.
Going deeper with convolutions
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich · 2015
Cited alongside, same era.
Layer normalization
Original
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton · 2016
Cited alongside, same era.
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2016
Cited alongside, same era.
Breaking the curse of dimensionality with convex neural networks
Francis Bach · 2017
Cited alongside, same era.
Graph signal recovery via primal-dual algorithms for total variation minimization
Peter Berger, Gabor Hannak, and Gerald Matz · 2017
Cited alongside, same era.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin · 2017
Cited alongside, same era.
Adaptive input representations for neural language modeling
Original
Alexei Baevski and Michael Auli · 2018
Cited alongside, same era.
Neural ordinary differential equations
Original
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud · 2018
Cited alongside, same era.
Bert: Pre-training of deep bidirectional transformers for language understanding
Original
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova · 2018
Cited alongside, same era.
Non-local neural networks
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He · 2018
Cited alongside, same era.

Then

Query-key normalization for transformers
Alex Henry, Prudhvi Raj Dachapally, Shubham Shantaram Pawar, and Yuxuan Chen · 2020
Later among the works it cites.
A proof that rectified deep neural networks overcome the curse of dimensionality in the numerical approximation of semilinear heat equations
Martin Hutzenthaler, Arnulf Jentzen, Thomas Kruse, and Tuan Anh Nguyen · 2020
Later among the works it cites.
On layer normalization in the transformer architecture
Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu · 2020
Later among the works it cites.
Finite neuron method and convergence analysis
Jinchao Xu · 2020
Later among the works it cites.
BEiT: BERT pre-training of image transformers
Hangbo Bao, Li Dong, and Furu Wei · 2021
Later among the works it cites.
Choose a transformer: Fourier or galerkin
Shuhao Cao · 2021
Later among the works it cites.
Emerging properties in self-supervised vision transformers
Original
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin · 2021
Later among the works it cites.
An empirical study of training self-supervised vision transformers
Original
Xinlei Chen, Saining Xie, and Kaiming He · 2021
Later among the works it cites.
Deepgreen: deep learning of green’s functions for nonlinear boundary value problems
Craig R Gin, Daniel E Shea, Steven L Brunton, and J Nathan Kutz · 2021
Later among the works it cites.
Transformer in transformer
Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang · 2021
Later among the works it cites.
Masked autoencoders are scalable vision learners
Original
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick · 2021
Later among the works it cites.
Swin transformer: Hierarchical vision transformer using shifted windows
Original
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo · 2021
Later among the works it cites.
Swin transformer: Hierarchical vision transformer using shifted windows
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo · 2021
Later among the works it cites.
Do vision transformers see like convolutional neural networks?
Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy · 2021
Later among the works it cites.
Training data-efficient image transformers & distillation through attention
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou · 2021
Later among the works it cites.
Multigraph transformer for free-hand sketch recognition
Peng Xu, Chaitanya K Joshi, and Xavier Bresson · 2021
Later among the works it cites.
Vision transformers are robust learners
Sayak Paul and Pin-Yu Chen · 2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…