A Review of automatic differentiation and its efficient implementation

Understand

Derivatives play a critical role in computational statistics, examples being Bayesian inference using Hamiltonian Monte Carlo sampling and the training of neural networks.

• Automatic differentiation is a powerful tool to automate the calculation of derivatives and is preferable to more traditional methods, especially when differentiating complex algorithms and mathematical functions.
• The implementation of automatic differentiation however requires some care to insure efficiency.
• Modern differentiation packages deploy a broad range of computational techniques to improve applicability, run time, and memory management.