2019

Mogrifier LSTM

Melis, Gábor, Kočiský, Tomáš, Blunsom, Phil

Understand

Many advances in Natural Language Processing have been based upon more expressive models for how inputs interact with the context in which they occur.

Recurrent networks, which have enjoyed a modicum of success, still lack the generalization and systematicity ultimately required for modelling language.
In this work, we propose an extension to the venerable Long Short-Term Memory in the form of mutual gating of the current input and the previous output.
This mechanism affords the modelling of a richer space of interactions between inputs and their context.

Built on

Learning representations by back-propagating errors
David E Rumelhart, Geoffrey E Hinton, Ronald J Williams, et al · 1988
Earlier work this paper cites.
Finding structure in time
Jeffrey L Elman · 1990
Earlier work this paper cites.
Backpropagation through time: what it does and how to do it
Paul J Werbos et al · 1990
Earlier work this paper cites.
Building a large annotated corpus of english: The Penn treebank
Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini · 1993
Earlier work this paper cites.
Lstm can solve hard long time lag problems
Sepp Hochreiter and Jürgen Schmidhuber · 1997
Earlier work this paper cites.
Reinforcement learning with long short-term memory
Bram Bakker · 2002
Earlier work this paper cites.
A system for robotic heart surgery that learns to tie knots using recurrent neural networks
Hermann Mayer, Faustino Gomez, Daan Wierstra, Istvan Nagy, Alois Knoll, and Jürgen Schmidhuber · 2008
Earlier work this paper cites.
Recurrent neural network based language model
Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernockỳ, and Sanjeev Khudanpur · 2010
Earlier work this paper cites.
Generating text with recurrent neural networks
Ilya Sutskever, James Martens, and Geoffrey E Hinton · 2011
Earlier work this paper cites.
The human knowledge compression contest
Marcus Hutter · 2012
Earlier work this paper cites.
On the difficulty of training recurrent neural networks
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio · 2013
Earlier work this paper cites.
Neural machine translation by jointly learning to align and translate
Original
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio · 2014
Earlier work this paper cites.
Adam: A method for stochastic optimization
Original
Diederik Kingma and Jimmy Ba · 2014
Earlier work this paper cites.
Long short-term memory based recurrent neural network architectures for large vocabulary speech recognition
Original
Hasim Sak, Andrew W. Senior, and Françoise Beaufays · 2014
Earlier work this paper cites.
Sequence to sequence learning with neural networks
Ilya Sutskever, Oriol Vinyals, and Quoc V Le · 2014
Earlier work this paper cites.
Gated feedback recurrent neural networks
Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio · 2015
Earlier work this paper cites.
Neural machine translation of rare words with subword units
Original
Rico Sennrich, Barry Haddow, and Alexandra Birch · 2015
Earlier work this paper cites.

Similar

Analyzing the behavior of visual question answering models
Original
Aishwarya Agrawal, Dhruv Batra, and Devi Parikh · 2016
Cited alongside, same era.
Unitary evolution recurrent neural networks
Martin Arjovsky, Amar Shah, and Yoshua Bengio · 2016
Cited alongside, same era.
A theoretically grounded application of dropout in recurrent neural networks
Yarin Gal and Zoubin Ghahramani · 2016
Cited alongside, same era.
Hypernetworks
Original
David Ha, Andrew Dai, and Quoc V Le · 2016
Cited alongside, same era.
Tying word vectors and word classifiers: A loss framework for language modeling
Original
Hakan Inan, Khashayar Khosravi, and Richard Socher · 2016
Cited alongside, same era.
Multiplicative LSTM for sequence modelling
Original
Ben Krause, Liang Lu, Iain Murray, and Steve Renals · 2016
Cited alongside, same era.
Assessing the ability of lstms to learn syntax-sensitive dependencies
Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg · 2016
Cited alongside, same era.
Pointer sentinel mixture models
Original
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher · 2016
Cited alongside, same era.
Using the output embedding to improve language models
Original
Ofir Press and Lior Wolf · 2016
Cited alongside, same era.
On multiplicative integration with recurrent neural networks
Yuhuai Wu, Saizheng Zhang, Ying Zhang, Yoshua Bengio, and Ruslan R Salakhutdinov · 2016
Cited alongside, same era.
Synthetic and natural noise both break neural machine translation
Original
Yonatan Belinkov and Yonatan Bisk · 2017
Cited alongside, same era.
Input switched affine networks: An rnn architecture designed for interpretability
Jakob N Foerster, Justin Gilmer, Jascha Sohl-Dickstein, Jan Chorowski, and David Sussillo · 2017
Cited alongside, same era.

Then

Regularizing and optimizing lstm language models
Original
Stephen Merity, Nitish Shirish Keskar, and Richard Socher · 2017
Later among the works it cites.
Lexical features in coreference resolution: To be used with caution
Original
Nafise Sadat Moosavi and Michael Strube · 2017
Later among the works it cites.
Sequence modeling via segmentations
Chong Wang, Yining Wang, Po-Sen Huang, Abdelrahman Mohamed, Dengyong Zhou, and Li Deng · 2017
Later among the works it cites.
Breaking the softmax bottleneck: a high-rank rnn language model
Original
Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, and William W Cohen · 2017
Later among the works it cites.
Neural architecture search with reinforcement learning
Original
Barret Zoph and Quoc V. Le · 2017
Later among the works it cites.
Trellis networks for sequence modeling
Original
Shaojie Bai, J Zico Kolter, and Vladlen Koltun · 2018
Later among the works it cites.
Bert: Pre-training of deep bidirectional transformers for language understanding
Original
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova · 2018
Later among the works it cites.
Frage: frequency-agnostic word representation
Chengyue Gong, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tie-Yan Liu · 2018
Later among the works it cites.
Universal language model fine-tuning for text classification
Original
Jeremy Howard and Sebastian Ruder · 2018
Later among the works it cites.
Adversarial example generation with syntactically controlled paraphrase networks
Original
Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer · 2018
Later among the works it cites.
Lstms can learn syntax-sensitive dependencies well, but modeling structure makes them better
Adhiguna Kuncoro, Chris Dyer, John Hale, Dani Yogatama, Stephen Clark, and Phil Blunsom · 2018
Later among the works it cites.
Pushing the bounds of dropout
Original
Gábor Melis, Charles Blundell, Tomáš Kočiskỳ, Karl Moritz Hermann, Chris Dyer, and Phil Blunsom · 2018
Later among the works it cites.
An analysis of neural language modeling at multiple scales
Original
Stephen Merity, Nitish Shirish Keskar, and Richard Socher · 2018
Later among the works it cites.
Deep contextualized word representations
Original
Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer · 2018
Later among the works it cites.
Transformer-xl: Attentive language models beyond a fixed-length context
Original
Zihang Dai, Zhilin Yang, Yiming Yang, William W Cohen, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov · 2019
Closest in time.
Dynamic evaluation of transformer language models
Original
Ben Krause, Emmanuel Kahembwe, Iain Murray, and Steve Renals · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…