Training DNNs in O(1) memory with MEM-DFA using Random Matrices

Understand

This work presents a method for reducing memory consumption to a constant complexity when training deep neural networks.

• The algorithm is based on the more biologically plausible alternatives of the backpropagation (BP): direct feedback alignment (DFA) and feedback alignment (FA), which use random matrices to propagate error.
• The proposed method, memory-efficient direct feedback alignment (MEM-DFA), uses higher independence of layers in DFA and allows avoiding storing at once all activation vectors, unlike standard BP, FA, and DFA.
• Thus, our algorithm's memory usage is constant regardless of the number of layers in a neural network.