Bidirectionally Self-Normalizing Neural Networks

Understand

The problem of vanishing and exploding gradients has been a long-standing obstacle that hinders the effective training of neural networks.

• Despite various tricks and techniques that have been employed to alleviate the problem in practice, there still lacks satisfactory theories or provable solutions.
• In this paper, we address the problem from the perspective of high-dimensional probability theory.
• We provide a rigorous result that shows, under mild conditions, how the vanishing/exploding gradients problem disappears with high probability if the neural networks have sufficient width.