Stochastic Bound Majorization

Understand

Recently a majorization method for optimizing partition functions of log-linear models was proposed alongside a novel quadratic variational upper-bound.

• In the batch setting, it outperformed state-of-the-art first- and second-order optimization methods on various learning tasks.
• We propose a stochastic version of this bound majorization method as well as a low-rank modification for high-dimensional data-sets.
• The resulting stochastic second-order method outperforms stochastic gradient descent (across variations and various tunings) both in terms of the number of iterations and computation time till convergence while finding a better quality parameter setting.