FastGRNN: A Fast, Accurate, Stable and Tiny Kilobyte Sized Gated Recurrent Neural Network

Understand

This paper develops the FastRNN and FastGRNN algorithms to address the twin RNN limitations of inaccurate training and inefficient prediction.

• Previous approaches have improved accuracy at the expense of prediction costs making them infeasible for resource-constrained and real-time applications.
• Unitary RNNs have increased accuracy somewhat by restricting the range of the state transition matrix's singular values but have also increased the model size as they require a larger number of hidden units to make up for the loss in expressive power.
• Gated RNNs have obtained state-of-the-art accuracies by adding extra parameters thereby resulting in even larger models.