Rethinking Full Connectivity in Recurrent Neural Networks

Understand

Recurrent neural networks (RNNs) are omnipresent in sequence modeling tasks.

• Practical models usually consist of several layers of hundreds or thousands of neurons which are fully connected.
• This places a heavy computational and memory burden on hardware, restricting adoption in practical low-cost and low-power devices.
• Compared to fully convolutional models, the costly sequential operation of RNNs severely hinders performance on parallel hardware.