Lifted Proximal Operator Machines

Understand

We propose a new optimization method for training feed-forward neural networks.

• By rewriting the activation function as an equivalent proximal operator, we approximate a feed-forward neural network by adding the proximal operators to the objective function as penalties, hence we call the lifted proximal operator machine (LPOM).
• LPOM is block multi-convex in all layer-wise weights and activations.
• This allows us to use block coordinate descent to update the layer-wise weights and activations in parallel.