A Rainbow in Deep Network Black Boxes

Understand

A central question in deep learning is to understand the functions learned by deep networks.

• What is their approximation class? Do the learned weights and representations depend on initialization? Previous empirical work has evidenced that kernels defined by network activations are similar across initializations.
• For shallow networks, this has been theoretically studied with random feature models, but an extension to deep networks has remained elusive.
• Here, we provide a deep extension of such random feature models, which we call the rainbow model.