Empirical Bounds on Linear Regions of Deep Rectifier Networks

Understand

We can compare the expressiveness of neural networks that use rectified linear units (ReLUs) by the number of linear regions, which reflect the number of pieces of the piecewise linear functions modeled by such networks.

• However, enumerating these regions is prohibitive and the known analytical bounds are identical for networks with same dimensions.
• In this work, we approximate the number of linear regions through empirical bounds based on features of the trained network and probabilistic inference.
• Our first contribution is a method to sample the activation patterns defined by ReLUs using universal hash functions.