Vector-output ReLU Neural Network Problems are Copositive Programs: Convex Analysis of Two Layer Networks and Polynomial-time Algorithms

Understand

We describe the convex semi-infinite dual of the two-layer vector-output ReLU neural network training problem.

• This semi-infinite dual admits a finite dimensional representation, but its support is over a convex set which is difficult to characterize.
• In particular, we demonstrate that the non-convex neural network training problem is equivalent to a finite-dimensional convex copositive program.
• Our work is the first to identify this strong connection between the global optima of neural networks and those of copositive programs.