2018

The unreasonable effectiveness of the forget gate

van der Westhuizen, Jos, Lasenby, Joan

Understand

Given the success of the gated recurrent unit, a natural question is whether all the gates of the long short-term memory (LSTM) network are necessary.

Previous research has shown that the forget gate is one of the most important gates in the LSTM.
Here we show that a forget-gate-only version of the LSTM with chrono-initialized biases, not only provides computational savings but outperforms the standard LSTM on multiple benchmark datasets and competes with some of the best contemporary models.
Our proposed network, the JANET, achieves accuracies of 99% and 92.5% on the MNIST and pMNIST datasets, outperforming the standard LSTM which yields accuracies of 98.5% and 91%.

Built on

Learning long-term dependencies in NARX recurrent neural networks
Lin, T., Horne, B. G., Tino, P., and Giles, C. L. (1996) · 1996
Earlier work this paper cites.
Long Short-Term Memory
Hochreiter, S. and Schmidhuber, J. (1997) · 1997
Earlier work this paper cites.
The MNIST Database of Handwritten Digits
LeCun, Y. A. (1998) · 1998
Earlier work this paper cites.
Recurrent nets that time and count
Gers, F. A. and Schmidhuber, J. (2000) · 2000
Earlier work this paper cites.
Learning to Forget: Continual Prediction with LSTM
Gers, F. A., Schmidhuber, J., and Cummins, F. (2000) · 2000
Earlier work this paper cites.
Physiobank, physiotoolkit, and physionet
Goldberger, A. L., Amaral, L. A., Glass, L., Hausdorff, J. M., Ivanov, P. C., Mark, R. G., Mietus, J. E., Moody, G. B., Peng, C., and Stanley, H. E. (2000) · 2000
Earlier work this paper cites.
The impact of the MIT-BIH Arrhythmia Database
Moody, G. B. and Mark, R. G. (2001) · 2001
Earlier work this paper cites.
Open source ECG analysis
Hamilton, P. S. (2002) · 2002
Earlier work this paper cites.
Tutorial on Training Recurrent Neural Networks, Covering BPPT, RTRL, EKF and the “Echo State Network” Approach
Jaeger, H. (2002) · 2002
Earlier work this paper cites.
Understanding the difficulty of training deep feedforward neural networks
Glorot, X. and Bengio, Y. (2010) · 2010
Earlier work this paper cites.
Practical Variational Inference for Neural Networks
Graves, A. (2011) · 2011
Earlier work this paper cites.
Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., and Bengio, Y. (2014) · 2014
Earlier work this paper cites.
TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., S. Corrado, G., Davis, A., Dean, J., Devin, M., Ghemawat, S., Goodfellow, I., Harp, A., Irving, G., Isard, M., Jia, Y., Jozefowicz, R., Kaiser, L., Kudlur, M., Levenberg, J., Mané, D., Monga, R., Moore, S., Murray, D., Olah, C., Schuster, M., Shlens, J., Steiner, B., Sutskever, I., Talwar, K., Tucker, P., Vanhoucke, V., Vasudevan, V., Viégas, F., Vinyals, O., Warden, P., Wattenberg, M., Wicke, M., Yu, Y., and Zheng, X. (2015) · 2015
Earlier work this paper cites.
Natural Language Understanding with Distributed Representation
Original
Cho, K. (2015) · 2015
Earlier work this paper cites.

Similar

LSTM: A Search Space Odyssey
Original
Greff, K., Srivastava, R. K., Koutník, J., Steunebrink, B. R., and Schmidhuber, J. (2015) · 2015
Cited alongside, same era.
Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
Han, S., Mao, H., and Dally, W. J. (2015) · 2015
Cited alongside, same era.
Distilling the Knowledge in a Neural Network
Hinton, G. E., Vinyals, O., and Dean, J. (2015) · 2015
Cited alongside, same era.
An empirical exploration of recurrent network architectures
Jozefowicz, R., Zaremba, W., and Sutskever, I. (2015) · 2015
Cited alongside, same era.
Adam: A Method for Stochastic Optimization
Kingma, D. and Ba, J. (2015) · 2015
Cited alongside, same era.
A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
Original
Le, Q. V., Jaitly, N., and Hinton, G. E. (2015) · 2015
Cited alongside, same era.
A Critical Review of Recurrent Neural Networks for Sequence Learning
Original
Lipton, Z. C., Berkowitz, J., and Elkan, C. (2015) · 2015
Cited alongside, same era.
Structured Transforms for Small-Footprint Deep Learning
Sindhwani, V., Sainath, T., and Kumar, S. (2015) · 2015
Cited alongside, same era.
Training Very Deep Networks
Srivastava, R. K., Greff, K., and Schmidhuber, J. (2015) · 2015
Cited alongside, same era.
Depth-gated Recurrent Neural Networks
Original
Yao, K., Cohn, T., Vylomova, K., Duh, K., and Dyer, C. (2015) · 2015
Cited alongside, same era.
Beyond CPU: Considering memory power consumption of software
Acar, H., Alptekin, G. I., Gelas, J. P., and Ghodous, P. (2016) · 2016
Cited alongside, same era.
Fathom: Reference workloads for modern deep learning methods
Adolf, R., Rama, S., Reagen, B., Wei, G., and Brooks, D. (2016) · 2016
Cited alongside, same era.

Then

Sequential Neural Models with Stochastic Layers
Fraccaro, M., Sø nderby, S. r. K., Paquet, U., and Winther, O. (2016) · 2016
Later among the works it cites.
Deep Learning
Goodfellow, I., Bengio, Y., and Courville, A. (2016) · 2016
Later among the works it cites.
Phased LSTM: Accelerating Recurrent Network Training for Long or Event-based Sequences
Neil, D., Pfeiffer, M., and Liu, S. (2016) · 2016
Later among the works it cites.
WaveNet: A Generative Model for Raw Audio
Original
Van Den Oord, A., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., and Kavukcuoglu, K. (2016) · 2016
Later among the works it cites.
Investigating gated recurrent networks for speech synthesis
Wu, Z. and King, S. (2016) · 2016
Later among the works it cites.
Architectural Complexity Measures of Recurrent Neural Networks
Zhang, S., Wu, Y., Che, T., Lin, Z., Memisevic, R., Salakhutdinov, R. R., and Bengio, Y. (2016) · 2016
Later among the works it cites.
Minimal gated unit for recurrent neural networks
Original
Zhou, G.-B., Wu, J., Zhang, C., and Zhou, Z. (2016) · 2016
Later among the works it cites.
Dilated Recurrent Neural Networks
Chang, S., Zhang, Y., Han, W., Yu, M., Guo, X., Tan, W., Cui, X., Witbrock, M., Hasegawa-Johnson, M. A., and Huang, T. S. (2017) · 2017
Later among the works it cites.
Wider and Deeper, Cheaper and Faster: Tensorized LSTMs for Sequence Learning
He, Z., Gao, S., Xiao, L., Liu, D., He, H., and Barber, D. (2017) · 2017
Later among the works it cites.
Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations
Krueger, D., Maharaj, T., Kramár, J., Pezeshki, M., Ballas, N., Ke, N. R., Goyal, A., Bengio, Y., Courville, A., and Pal, C. (2017) · 2017
Later among the works it cites.
Learning Simpler Language Models with the Differential State Framework
Ororbia II, A. G., Mikolov, T., and Reitter, D. (2017) · 2017
Later among the works it cites.
Accelerating Recurrent Neural Networks: A Memory-Efficient Approach
Wang, Z., Lin, J., and Wang, Z. (2017) · 2017
Later among the works it cites.
BioSPPy: Biosignal processing in Python
Carreiras, C., Alves, A. P., Lourenço, A., Canento, F., Silva, H., Fred, A., et al. (2015) · 2018
Closest in time.
Can recurrent neural networks warp time?
Tallec, C. and Ollivier, Y. (2018) · 2018
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…