Semi-groups of stochastic gradient descent and online principal component analysis: properties and diffusion approximations

Understand

We study the Markov semigroups for two important algorithms from machine learning: stochastic gradient descent (SGD) and online principal component analysis (PCA).

• We investigate the effects of small jumps on the properties of the semi-groups.
• Properties including regularity preserving, $L^{\infty}$ contraction are discussed.
• These semigroups are the dual of the semigroups for evolution of probability, while the latter are $L^{1}$ contracting and positivity preserving.