MicroAdam: Accurate Adaptive Optimization with Low Space Overhead and Provable Convergence

Understand

We propose a new variant of the Adam optimizer called MicroAdam that specifically minimizes memory overheads, while maintaining theoretical convergence guarantees.

• We achieve this by compressing the gradient information before it is fed into the optimizer state, thereby reducing its memory footprint significantly.
• We control the resulting compression error via a novel instance of the classical \emph{error feedback} mechanism from distributed optimization in which *the error correction information is itself compressed* to allow for practical memory gains.
• We prove that the resulting approach maintains theoretical convergence guarantees competitive to those of AMSGrad, while providing good practical performance.