Fundamental Limits of Two-layer Autoencoders, and Achieving Them with Gradient Methods

Understand

Autoencoders are a popular model in many branches of machine learning and lossy data compression.

• However, their fundamental limits, the performance of gradient methods and the features learnt during optimization remain poorly understood, even in the two-layer setting.
• In fact, earlier work has considered either linear autoencoders or specific training regimes (leading to vanishing or diverging compression rates).
• Our paper addresses this gap by focusing on non-linear two-layer autoencoders trained in the challenging proportional regime in which the input dimension scales linearly with the size of the representation.