Measuring the Algorithmic Efficiency of Neural Networks

Understand

Three factors drive the advance of AI: algorithmic innovation, data, and the amount of compute available for training.

• Algorithmic progress has traditionally been more difficult to quantify than compute and data.
• In this work, we argue that algorithmic progress has an aspect that is both straightforward to measure and interesting: reductions over time in the compute needed to reach past capabilities.
• We show that the number of floating-point operations required to train a classifier to AlexNet-level performance on ImageNet has decreased by a factor of 44x between 2012 and 2019.