How does training shape the Riemannian geometry of neural network representations?

Understand

In machine learning, there is a long history of trying to build neural networks that can learn from fewer example data by baking in strong geometric priors.

• However, it is not always clear a priori what geometric constraints are appropriate for a given task.
• Here, we explore the possibility that one can uncover useful geometric inductive biases by studying how training molds the Riemannian geometry induced by unconstrained neural network feature maps.
• We first show that at infinite width, neural networks with random parameters induce highly symmetric metrics on input space.