Memorization in Overparameterized Autoencoders

Understand

The ability of deep neural networks to generalize well in the overparameterized regime has become a subject of significant research interest.

• We show that overparameterized autoencoders exhibit memorization, a form of inductive bias that constrains the functions learned through the optimization process to concentrate around the training examples, although the network could in principle represent a much larger function class.
• In particular, we prove that single-layer fully-connected autoencoders project data onto the (nonlinear) span of the training examples.
• In addition, we show that deep fully-connected autoencoders learn a map that is locally contractive at the training examples, and hence iterating the autoencoder results in convergence to the training examples.