Loss Landscape Dependent Self-Adjusting Learning Rates in Decentralized Stochastic Gradient Descent

Understand

Distributed Deep Learning (DDL) is essential for large-scale Deep Learning (DL) training.

• Synchronous Stochastic Gradient Descent (SSGD) 1 is the de facto DDL optimization method.
• Using a sufficiently large batch size is critical to achieving DDL runtime speedup.
• In a large batch setting, the learning rate must be increased to compensate for the reduced number of parameter updates.