Memory-Efficient Adaptive Optimization

Understand

Adaptive gradient-based optimizers such as Adagrad and Adam are crucial for achieving state-of-the-art performance in machine translation and language modeling.

• However, these methods maintain second-order statistics for each parameter, thus introducing significant memory overheads that restrict the size of the model being used as well as the number of examples in a mini-batch.
• We describe an effective and flexible adaptive optimization method with greatly reduced memory overhead.
• Our method retains the benefits of per-parameter adaptivity while allowing significantly larger models and batch sizes.