Gradients are Not All You Need

Understand

Differentiable programming techniques are widely used in the community and are responsible for the machine learning renaissance of the past several decades.

• While these methods are powerful, they have limits.
• In this short report, we discuss a common chaos based failure mode which appears in a variety of differentiable circumstances, ranging from recurrent neural networks and numerical physics simulation to training learned optimizers.
• We trace this failure to the spectrum of the Jacobian of the system under study, and provide criteria for when a practitioner might expect this failure to spoil their differentiation based optimization algorithms.