Dynamic Memory Based Adaptive Optimization

Understand

Define an optimizer as having memory $k$ if it stores $k$ dynamically changing vectors in the parameter space.

• Classical SGD has memory $0$, momentum SGD optimizer has $1$ and Adam optimizer has $2$.
• We address the following questions: How can optimizers make use of more memory units? What information should be stored in them? How to use them for the learning steps? As an approach to the last question, we introduce a general method called "Retrospective Learning Law Correction" or shortly RLLC.
• This method is designed to calculate a dynamically varying linear combination (called learning law) of memory units, which themselves may evolve arbitrarily.