DrMAD: Distilling Reverse-Mode Automatic Differentiation for Optimizing Hyperparameters of Deep Neural Networks

Understand

The performance of deep neural networks is well-known to be sensitive to the setting of their hyperparameters.

• Recent advances in reverse-mode automatic differentiation allow for optimizing hyperparameters with gradients.
• The standard way of computing these gradients involves a forward and backward pass of computations.
• However, the backward pass usually needs to consume unaffordable memory to store all the intermediate variables to exactly reverse the forward training procedure.