Generalized Tangent Kernel: A Unified Geometric Foundation for Natural Gradient and Standard Gradient

Understand

Natural gradients have been widely studied from both theoretical and empirical perspectives, and it is commonly believed that natural gradients have advantages over standard (Euclidean) gradients in capturing the intrinsic geometric structure of the underlying function space and being invariant under reparameterization.

• However, for function optimization, a fundamental theoretical issue regarding the existence of natural gradients on the function space remains underexplored.
• We address this issue by providing a geometric perspective and mathematical framework for studying both natural gradient and standard gradient that is more complete than existing studies.
• The key tool that unifies natural gradient and standard gradient is a generalized form of the Neural Tangent Kernel (NTK), which we name the Generalized Tangent Kernel (GTK).