Reconstruction of training samples from loss functions

Understand

This paper presents a new mathematical framework to analyze the loss functions of deep neural networks with ReLU functions.

• Furthermore, as as application of this theory, we prove that the loss functions can reconstruct the inputs of the training samples up to scalar multiplication (as vectors) and can provide the number of layers and nodes of the deep neural network.
• Namely, if we have all input and output of a loss function (or equivalently all possible learning process), for all input of each training sample $x_i \in \mathbb{R}^n$, we can obtain vectors $x'_i\in \mathbb{R}^n$ satisfying $x_i=c_ix'_i$ for some $c_i \neq 0$.
• To prove theorem, we introduce the notion of virtual polynomials, which are polynomials written as the output of a node in a deep neural network.