Data-driven Weight Initialization with Sylvester Solvers

Understand

In this work, we propose a data-driven scheme to initialize the parameters of a deep neural network.

• This is in contrast to traditional approaches which randomly initialize parameters by sampling from transformed standard distributions.
• Such methods do not use the training data to produce a more informed initialization.
• Our method uses a sequential layer-wise approach where each layer is initialized using its input activations.