Improving Convergence and Generalization Using Parameter Symmetries

Understand

In many neural networks, different values of the parameters may result in the same loss value.

• Parameter space symmetries are loss-invariant transformations that change the model parameters.
• Teleportation applies such transformations to accelerate optimization.
• However, the exact mechanism behind this algorithm's success is not well understood.