Re-parameterizing Your Optimizers rather than Architectures

Understand

The well-designed structures in neural networks reflect the prior knowledge incorporated into the models.

• However, though different models have various priors, we are used to training them with model-agnostic optimizers such as SGD.
• In this paper, we propose to incorporate model-specific prior knowledge into optimizers by modifying the gradients according to a set of model-specific hyper-parameters.
• Such a methodology is referred to as Gradient Re-parameterization, and the optimizers are named RepOptimizers.