Compute-Efficient Deep Learning: Algorithmic Trends and Opportunities

Understand

Although deep learning has made great progress in recent years, the exploding economic and environmental costs of training neural networks are becoming unsustainable.

• To address this problem, there has been a great deal of research on *algorithmically-efficient deep learning*, which seeks to reduce training costs not at the hardware or implementation level, but through changes in the semantics of the training program.
• In this paper, we present a structured and comprehensive overview of the research in this field.
• First, we formalize the *algorithmic speedup* problem, then we use fundamental building blocks of algorithmically efficient training to develop a taxonomy.