Every Model Learned by Gradient Descent Is Approximately a Kernel Machine

Understand

Deep learning's successes are often attributed to its ability to automatically discover new representations of the data, rather than relying on handcrafted features like other learning methods.

• We show, however, that deep networks learned by the standard gradient descent algorithm are in fact mathematically approximately equivalent to kernel machines, a learning method that simply memorizes the data and uses it directly for prediction via a similarity function (the kernel).
• This greatly enhances the interpretability of deep network weights, by elucidating that they are effectively a superposition of the training examples.
• The network architecture incorporates knowledge of the target function into the kernel.