AdaBatch: Adaptive Batch Sizes for Training Deep Neural Networks

Understand

Training deep neural networks with Stochastic Gradient Descent, or its variants, requires careful choice of both learning rate and batch size.

• While smaller batch sizes generally converge in fewer training epochs, larger batch sizes offer more parallelism and hence better computational efficiency.
• We have developed a new training approach that, rather than statically choosing a single batch size for all epochs, adaptively increases the batch size during the training process.
• Our method delivers the convergence rate of small batch sizes while achieving performance similar to large batch sizes.