On Multiplicative Integration with Recurrent Neural Networks

Understand

We introduce a general and simple structural design called Multiplicative Integration (MI) to improve recurrent neural networks (RNNs).

• MI changes the way in which information from difference sources flows and is integrated in the computational building block of an RNN, while introducing almost no extra parameters.
• The new structure can be easily embedded into many popular RNN models, including LSTMs and GRUs.
• We empirically analyze its learning behaviour and conduct evaluations on several tasks using different RNN models.