Fetching the paper…

Reading the bibliography…

2016

Forward-Mode Automatic Differentiation in Julia

Revels, Jarrett, Lubin, Miles, Papamarkou, Theodore

Original
PDF
Post on X

Understand

We present ForwardDiff, a Julia package for forward-mode automatic differentiation (AD) featuring performance competitive with low-level languages like C++.

Unlike recently developed AD tools in other popular high-level languages such as Python and MATLAB, ForwardDiff takes advantage of just-in-time (JIT) compilation to transparently recompile AD-unaware user code, enabling efficient support for higher-order differentiation and differentiation using custom number types (including complex numbers).
For gradient and Jacobian calculations, ForwardDiff provides a variant of vector-forward mode that avoids expensive heap allocation and makes better use of memory bandwidth than traditional vector mode.
In our numerical experiments, we demonstrate that for nontrivially large dimensions, ForwardDiff's gradient computations can be faster than a reverse-mode implementation from the Python-based autograd package.

Forward-Mode Automatic Differentiation in Julia · Around

Built on

AMPL: A modeling language for mathematical programming
Robert Fourer, David M Gay, and Brian W Kernighan · 2003
Earlier work this paper cites.
Perturbation confusion and referential transparency: Correct functional implementation of forward-mode AD
Jeffrey Mark Siskind and Barak A. Pearlmutter · 2005
Earlier work this paper cites.
Efficient computation of sparse Hessians using coloring and automatic differentiation
Assefaw H. Gebremedhin, Arijit Tarafdar, Alex Pothen, and Andrea Walther · 2009
Earlier work this paper cites.
Automatic differentiation through the use of hyper-dual numbers for second derivatives
Jeffrey A. Fike and Juan J. Alonso · 2012
Earlier work this paper cites.

Similar

Algorithmic differentiation in python with algopy
Sebastian F. Walter and Lutz Lehmann · 2013
Cited alongside, same era.
An efficient overloaded method for computing derivatives of mathematical functions in matlab
Michael A. Patterson, Matthew Weinstein, and Anil V. Rao · 2013
Cited alongside, same era.
Julia: A fresh approach to numerical computing
Jeff Bezanson, Alan Edelman, Stefan Karpinski, and Viral B. Shah · 2014
Cited alongside, same era.

Then

Gradient-based Hyperparameter Optimization through Reversible Learning
Dougal Maclaurin, David K. Duvenaud, and Ryan P. Adams · 2015
Later among the works it cites.
JuMP: A Modeling Language for Mathematical Optimization
I. Dunning, J. Huchette, and M. Lubin · 2015
Later among the works it cites.
A vector forward mode of automatic differentiation for generalized derivative evaluation
Kamil A. Khan and Paul I. Barton · 2015
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…