Optimizing Performance of Recurrent Neural Networks on GPUs

Understand

As recurrent neural networks become larger and deeper, training times for single networks are rising into weeks or even months.

• As such there is a significant incentive to improve the performance and scalability of these networks.
• While GPUs have become the hardware of choice for training and deploying recurrent models, the implementations employed often make use of only basic optimizations for these architectures.
• In this article we demonstrate that by exposing parallelism between operations within the network, an order of magnitude speedup across a range of network sizes can be achieved over a naive implementation.