MuFuRU: The Multi-Function Recurrent Unit

Understand

Recurrent neural networks such as the GRU and LSTM found wide adoption in natural language processing and achieve state-of-the-art results for many tasks.

• These models are characterized by a memory state that can be written to and read from by applying gated composition operations to the current input and the previous state.
• However, they only cover a small subset of potentially useful compositions.
• We propose Multi-Function Recurrent Units (MuFuRUs) that allow for arbitrary differentiable functions as composition operations.