Understand
Hyperparameter tuning, particularly the selection of an appropriate learning rate in adaptive gradient training methods, remains a challenge.
- To address this challenge, we propose a novel parameter-free optimizer, \textsc{AdamG} (Adam with the Golden step size), designed to automatically adapt to diverse optimization problems without task-specific learning-rate tuning.
- The core technique underlying \textsc{AdamG} is our golden step size derived for the AdaGrad-Norm algorithm, which is expected to help AdaGrad-Norm preserve tuning-free convergence and approximate the optimal step size in expectation across diverse optimization scenarios.
- To better evaluate tuning-free performance, we propose a novel evaluation criterion, \textit{reliability}, to comprehensively assess the efficacy of parameter-free optimizers in addition to classical performance criteria.