A Brief Introduction to Automatic Differentiation for Machine Learning

Understand

Machine learning and neural network models in particular have been improving the state of the art performance on many artificial intelligence related tasks.

• Neural network models are typically implemented using frameworks that perform gradient based optimization methods to fit a model to a dataset.
• These frameworks use a technique of calculating derivatives called automatic differentiation (AD) which removes the burden of performing derivative calculations from the model designer.
• In this report we describe AD, its motivations, and different implementation approaches.