Revisiting Activation Regularization for Language RNNs

Understand

Recurrent neural networks (RNNs) serve as a fundamental building block for many sequence tasks across natural language processing.

• Recent research has focused on recurrent dropout techniques or custom RNN cells in order to improve performance.
• Both of these can require substantial modifications to the machine learning model or to the underlying RNN configurations.
• We revisit traditional regularization techniques, specifically L2 regularization on RNN activations and slowness regularization over successive hidden states, to improve the performance of RNNs on the task of language modeling.