2017

Regularizing and Optimizing LSTM Language Models

Merity, Stephen, Keskar, Nitish Shirish, Socher, Richard

Understand

Recurrent neural networks (RNNs), such as long short-term memory networks (LSTMs), serve as a fundamental building block for many sequence learning tasks, including machine translation, language modeling, and question answering.

In this paper, we consider the specific problem of word-level language modeling and investigate strategies for regularizing and optimizing LSTM-based models.
We propose the weight-dropped LSTM which uses DropConnect on hidden-to-hidden weights as a form of recurrent regularization.
Further, we introduce NT-ASGD, a variant of the averaged stochastic gradient method, wherein the averaging trigger is determined using a non-monotonic condition as opposed to being tuned by the user.

Built on

Connectionist learning procedures
Hinton, G. E · 1989
Earlier work this paper cites.
Learning invariance from transformation sequences
Földiák, P · 1991
Earlier work this paper cites.
Acceleration of stochastic approximation by averaging
Polyak, B. and Juditsky, A · 1992
Earlier work this paper cites.
Moses: Open source toolkit for statistical machine translation
Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., Bertoldi, N., Cowan, B., Shen, W., Moran, C., Zens, R., Dyer, C., Bojar, O., Constantin, A., and Herbst, E · 2007
Earlier work this paper cites.
Recurrent neural network based language model
Mikolov, T., Karafiát, M., Burget, L., Cernocký, J., and Khudanpur, S · 2010
Earlier work this paper cites.
Adaptive subgradient methods for online learning and stochastic optimization
Duchi, J., Hazan, E., and Singer, Y · 2011
Earlier work this paper cites.
Low complexity proto-value function learning from sensory observations with incremental slow feature analysis
Luciw, M. and Schmidhuber, J · 2012
Earlier work this paper cites.
Context dependent recurrent neural network language model
Mikolov, T. and Zweig, G · 2012
Earlier work this paper cites.
Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude
Tieleman, T. and Hinton, G · 2012
Earlier work this paper cites.
On the importance of initialization and momentum in deep learning
Sutskever, I., Martens, J., Dahl, G., and Hinton, G · 2013
Earlier work this paper cites.
Regularization of neural networks using dropconnect
Wan, L., Zeiler, M., Zhang, S., LeCun, Y, and Fergus, R · 2013
Earlier work this paper cites.
Adam: A method for stochastic optimization
Original
Kingma, D. and Ba, J · 2014
Earlier work this paper cites.
Dropout: a simple way to prevent neural networks from overfitting
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R · 2014
Earlier work this paper cites.
Recurrent neural network regularization
Original
Zaremba, W., Sutskever, I., and Vinyals, O · 2014
Earlier work this paper cites.
Train faster, generalize better: Stability of stochastic gradient descent
Original
Hardt, M., Recht, B., and Singer, Y · 2015
Earlier work this paper cites.

Similar

Batch normalization: Accelerating deep network training by reducing internal covariate shift
Ioffe, S. and Szegedy, C · 2015
Cited alongside, same era.
Learning state representations with robotic priors
Jonschkowski, R. and Brock, O · 2015
Cited alongside, same era.
A nonmonotone learning rate strategy for sgd training of deep neural networks
Keskar, N. and Saon, G · 2015
Cited alongside, same era.
Unitary evolution recurrent neural networks
Arjovsky, M., Shah, A., and Bengio, Y · 2016
Cited alongside, same era.
Layer normalization
Original
Ba, J., Kiros, J., and Hinton, G. E · 2016
Cited alongside, same era.
Strongly-typed recurrent neural networks
Original
Balduzzi, D. and Ghifary, M · 2016
Cited alongside, same era.
Optimization methods for large-scale machine learning
Original
Bottou, L., Curtis, F. E., and Nocedal, J · 2016
Cited alongside, same era.
Quasi-Recurrent Neural Networks
Original
Bradbury, J., Merity, S., Xiong, C., and Socher, R · 2016
Cited alongside, same era.
Recurrent batch normalization
Original
Cooijmans, T., Ballas, N., Laurent, C., and Courville, A. C · 2016
Cited alongside, same era.
A theoretically grounded application of dropout in recurrent neural networks
Gal, Y. and Ghahramani, Z · 2016
Cited alongside, same era.
Improving neural language models with a continuous cache
Original
Grave, E., Joulin, A., and Usunier, N · 2016
Cited alongside, same era.
Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling
Original
Inan, H., Khosravi, K., and Socher, R · 2016
Cited alongside, same era.

Then

Pointer Sentinel Mixture Models
Original
Merity, S., Xiong, C., Bradbury, J., and Socher, R · 2016
Later among the works it cites.
Gradient descent converges to minimizers: The case of non-isolated critical points
Original
Panageas, I. and Piliouras, G · 2016
Later among the works it cites.
Using the output embedding to improve language models
Original
Press, O. and Wolf, L · 2016
Later among the works it cites.
Recurrent dropout without memory loss
Semeniuta, S., Severyn, A., and Barth, E · 2016
Later among the works it cites.
Query-Reduction Networks for Question Answering
Original
Seo, M., Min, S., Farhadi, A., and Hajishirzi, H · 2016
Later among the works it cites.
Full-capacity unitary recurrent neural networks
Wisdom, S., Powers, T., Hershey, J., Le Roux, J., and Atlas, L · 2016
Later among the works it cites.
Recurrent highway networks
Original
Zilly, J. G., Srivastava, R. K., Koutník, J., and Schmidhuber, J · 2016
Later among the works it cites.
Neural architecture search with reinforcement learning
Original
Zoph, B. and Le, Q. V · 2016
Later among the works it cites.
Accurate, large minibatch sgd: Training imagenet in 1 hour
Original
Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., and He, K · 2017
Closest in time.
Stochastic gradient descent as approximate bayesian inference
Original
Mandt, S., Hoffman, M. D., and Blei, D. M · 2017
Closest in time.
On the State of the Art of Evaluation in Neural Language Models
Original
Melis, G., Dyer, C., and Blunsom, P · 2017
Closest in time.
Revisiting activation regularization for language rnns
Original
Merity, S., McCann, B., and Socher, R · 2017
Closest in time.
Unbiasing truncated backpropagation through time
Original
Tallec, C. and Ollivier, Y · 2017
Closest in time.
The marginal value of adaptive gradient methods in machine learning
Original
Wilson, A. C, Roelofs, R., Stern, M., Srebro, N., and Recht, B · 2017
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…