Improving Optimization for Models With Continuous Symmetry Breaking

Understand

Many loss functions in representation learning are invariant under a continuous symmetry transformation.

• For example, the loss function of word embeddings (Mikolov et al., 2013) remains unchanged if we simultaneously rotate all word and context embedding vectors.
• We show that representation learning models for time series possess an approximate continuous symmetry that leads to slow convergence of gradient descent.
• We propose a new optimization algorithm that speeds up convergence using ideas from gauge theory in physics.