The simple essence of automatic differentiation

Understand

Automatic differentiation (AD) in reverse mode (RAD) is a central component of deep learning and other uses of large-scale optimization.

• Commonly used RAD algorithms such as backpropagation, however, are complex and stateful, hindering deep understanding, improvement, and parallel execution.
• This paper develops a simple, generalized AD algorithm calculated from a simple, natural specification.
• The general algorithm is then specialized by varying the representation of derivatives.