Degrees of Freedom in Deep Neural Networks

Understand

In this paper, we explore degrees of freedom in deep sigmoidal neural networks.

• We show that the degrees of freedom in these models is related to the expected optimism, which is the expected difference between test error and training error.
• We provide an efficient Monte-Carlo method to estimate the degrees of freedom for multi-class classification methods.
• We show degrees of freedom are lower than the parameter count in a simple XOR network.