2018

Learning Longer-term Dependencies in RNNs with Auxiliary Losses

Trinh, Trieu H., Dai, Andrew M., Luong, Minh-Thang et al.

Understand

Despite recent advances in training recurrent neural networks (RNNs), capturing long-term dependencies in sequences remains a fundamental challenge.

Most approaches use backpropagation through time (BPTT), which is difficult to scale to very long sequences.
This paper proposes a simple method that improves the ability to capture long term dependencies in RNNs by adding an unsupervised auxiliary loss to the original objective.
This auxiliary loss forces RNNs to either reconstruct previous events or predict next events in a sequence, making truncated backpropagation feasible for long sequences and also improving full BPTT.

Built on

Parallel distributed processing: Explorations in the microstructure of cognition, vol. 1
Rumelhart, D. E., Hinton, G. E., and Williams, R. J · 1986
Earlier work this paper cites.
Learning complex, extended sequences using the principle of history compression
Schmidhuber, J · 1992
Earlier work this paper cites.
The Penn treebank: Annotating predicate argument structure
Marcus, M., Kim, G., Marcinkiewicz, M. A., MacIntyre, R., Bies, A., Ferguson, M., Katz, K., and Schasberger, B · 1994
Earlier work this paper cites.
Hierarchical recurrent neural networks for long-term dependencies
El Hihi, S. and Bengio, Y · 1996
Earlier work this paper cites.
Long short-term memory
Hochreiter, S. and Schmidhuber, J · 1997
Earlier work this paper cites.
Supervised neural networks for the classification of structures
Sperduti, A. and Starita, A · 1997
Earlier work this paper cites.
A general framework for adaptive processing of data structures
Frasconi, P., Gori, M., and Sperduti, A · 1998
Earlier work this paper cites.
Learning to forget: Continual prediction with LSTM
Gers, F. A., Schmidhuber, J., and Cummins, F · 1999
Earlier work this paper cites.
Gradient flow in recurrent nets: the difficulty of learning long-term dependencies
Hochreiter, S., Bengio, Y., Frasconi, P., and Schmidhuber, J · 2001
Earlier work this paper cites.
Short-term memory in orthogonal neural networks
White, O. L., Lee, D. D., and Sompolinsky, H · 2004
Earlier work this paper cites.
Novel dataset for fine-grained image categorization
Khosla, A., Jayadevaprakash, N., Yao, B., and Fei-Fei, L · 2011
Earlier work this paper cites.
Learning recurrent neural networks with hessian-free optimization
Martens, J. and Sutskever, I · 2011
Earlier work this paper cites.
Parsing natural scenes and natural language with recursive neural networks
Socher, R., Lin, C., Ng, A. Y., and Manning, C. D · 2011
Earlier work this paper cites.
Lecture 6.5—RmsProp: Divide the gradient by a running average of its recent magnitude
Tieleman, T. and Hinton, G · 2012
Earlier work this paper cites.
Generating sequences with recurrent neural networks
Original
Graves, A · 2013
Earlier work this paper cites.
On the difficulty of training recurrent neural networks
Pascanu, R., Mikolov, T., and Bengio, Y · 2013
Earlier work this paper cites.

Similar

Regularization of neural networks using dropconnect
Wan, L., Zeiler, M., Zhang, S., Le Cun, Y., and Fergus, R · 2013
Cited alongside, same era.
Learning phrase representations using RNN encoder-decoder for statistical machine translation
Cho, K., Van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., and Bengio, Y · 2014
Cited alongside, same era.
Empirical evaluation of gated recurrent neural networks on sequence modeling
Original
Chung, J., Gulcehre, C., Cho, K., and Bengio, Y · 2014
Cited alongside, same era.
Adam: A method for stochastic optimization
Kingma, D. P. and Ba, J · 2014
Cited alongside, same era.
A clockwork RNN
Koutnik, J., Greff, K., Gomez, F., and Schmidhuber, J · 2014
Cited alongside, same era.
Learning longer memory in recurrent neural networks
Original
Mikolov, T., Joulin, A., Chopra, S., Mathieu, M., and Ranzato, M · 2014
Cited alongside, same era.
Attention for fine-grained categorization
Original
Sermanet, P., Frome, A., and Real, E · 2014
Cited alongside, same era.
TensorFlow: Large-scale machine learning on heterogeneous systems, 2015
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., et al · 2015
Cited alongside, same era.
Neural machine translation by jointly learning to align and translate
Bahdanau, D., Cho, K., and Bengio, Y · 2015
Cited alongside, same era.
Semi-supervised sequence learning
Dai, A. M. and Le, Q. V · 2015
Cited alongside, same era.
A simple way to initialize recurrent networks of rectified linear units
Original
Le, Q. V., Jaitly, N., and Hinton, G. E · 2015
Cited alongside, same era.
Effective approaches to attention-based neural machine translation
Luong, M.-T., Pham, H., and Manning, C. D · 2015
Cited alongside, same era.

Then

Very deep convolutional networks for natural language processing
Original
Conneau, A., Schwenk, H., Barrault, L., and LeCun, Y · 2016
Later among the works it cites.
Memory-efficient backpropagation through time
Gruslys, A., Munos, R., Danihelka, I., Lanctot, M., and Graves, A · 2016
Later among the works it cites.
Deep residual learning for image recognition
He, K., Zhang, X., Ren, S., and Sun, J · 2016
Later among the works it cites.
Recurrent orthogonal networks and long-memory tasks
Henaff, M., Szlam, A., and LeCun, Y · 2016
Later among the works it cites.
On multiplicative integration with recurrent neural networks
Wu, Y., Zhang, S., Zhang, Y., Bengio, Y., and Salakhutdinov, R. R · 2016
Later among the works it cites.
Efficient character-level document classification by combining convolution and recurrent layers
Original
Xiao, Y. and Cho, K · 2016
Later among the works it cites.
Z-forcing: Training stochastic recurrent networks
GOYAL, A. G. A. P., Sordoni, A., Côté, M.-A., Ke, N., and Bengio, Y · 2017
Later among the works it cites.
Decoupled neural interfaces using synthetic gradients
Jaderberg, M., Czarnecki, W. M., Osindero, S., Vinyals, O., Graves, A., and Kavukcuoglu, K · 2017
Later among the works it cites.
Unsupervised pretraining for sequence to sequence learning
Ramachandran, P., Liu, P. J., and Le, Q. V · 2017
Later among the works it cites.
PixelCNN++: Improving the PixelCNN with discretized logistic mixture likelihood and other modifications
Salimans, T., Karpathy, A., Chen, X., and Kingma, D. P · 2017
Later among the works it cites.
Attention is all you need
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I · 2017
Later among the works it cites.
Learning to skim text
Yu, A. W., Lee, H., and Le, Q. V · 2017
Later among the works it cites.
Recurrent highway networks
Zilly, J. G., Srivastava, R. K., Koutník, J., and Schmidhuber, J · 2017
Later among the works it cites.
Skip RNN: Learning to skip state updates in recurrent neural networks
Campos, V., Jou, B., Giró-i Nieto, X., Torres, J., and Chang, S.-F · 2018
Closest in time.
Generating Wikipedia by summarizing long sequences
Liu, P. J., Saleh, M., Pot, E., Goodrich, B., Sepassi, R., Kaiser, L., and Shazeer, N · 2018
Closest in time.
Neural speed reading via skim-rnn
Seo, M., Min, S., Farhadi, A., and Hajishirzi, H · 2018
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…