A Library of Mirrors: Deep Neural Nets in Low Dimensions are Convex Lasso Models with Reflection Features

Understand

We prove that training neural networks on 1-D data is equivalent to solving convex Lasso problems with discrete, explicitly defined dictionary matrices.

• We consider neural networks with piecewise linear activations and depths ranging from 2 to an arbitrary but finite number of layers.
• We first show that two-layer networks with piecewise linear activations are equivalent to Lasso models using a discrete dictionary of ramp functions, with breakpoints corresponding to the training data points.
• In certain general architectures with absolute value or ReLU activations, a third layer surprisingly creates features that reflect the training data about themselves.