Neural Networks Fail to Learn Periodic Functions and How to Fix It

Understand

Previous literature offers limited clues on how to learn a periodic function using modern neural networks.

• We start with a study of the extrapolation properties of neural networks; we prove and demonstrate experimentally that the standard activations functions, such as ReLU, tanh, sigmoid, along with their variants, all fail to learn to extrapolate simple periodic functions.
• We hypothesize that this is due to their lack of a "periodic" inductive bias.
• As a fix of this problem, we propose a new activation, namely, $x + \sin^2(x)$, which achieves the desired periodic inductive bias to learn a periodic function while maintaining a favorable optimization property of the ReLU-based activations.