2018

Trellis Networks for Sequence Modeling

Bai, Shaojie, Kolter, J. Zico, Koltun, Vladlen

Understand

We present trellis networks, a new architecture for sequence modeling.

On the one hand, a trellis network is a temporal convolutional network with special structure, characterized by weight tying across depth and direct injection of the input into deep layers.
On the other hand, we show that truncated recurrent networks are equivalent to trellis networks with special sparsity structure in their weight matrices.
Thus trellis networks with general weight matrices generalize truncated recurrent networks.

Built on

Backpropagation applied to handwritten zip code recognition
Yann LeCun, Bernhard Boser, John S. Denker, Donnie Henderson, Richard E. Howard, Wayne Hubbard, and Lawrence D. Jackel · 1989
Earlier work this paper cites.
Phoneme recognition using time-delay neural networks
Alex Waibel, Toshiyuki Hanazawa, Geoffrey Hinton, Kiyohiro Shikano, and Kevin J Lang · 1989
Earlier work this paper cites.
Finding structure in time
Jeffrey L Elman · 1990
Earlier work this paper cites.
Backpropagation through time: What it does and how to do it
Paul J Werbos · 1990
Earlier work this paper cites.
Building a large annotated corpus of English: The Penn treebank
Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini · 1993
Earlier work this paper cites.
Long short-term memory
Sepp Hochreiter and Jürgen Schmidhuber · 1997
Earlier work this paper cites.
Learning multiple layers of features from tiny images
Alex Krizhevsky and Geoffrey Hinton · 2009
Earlier work this paper cites.
Recurrent neural network based language model
Tomas Mikolov, Martin Karafiát, Lukás Burget, Jan Cernocký, and Sanjeev Khudanpur · 2010
Earlier work this paper cites.
Natural language processing (almost) from scratch
Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel P. Kuksa · 2011
Earlier work this paper cites.
Generating text with recurrent neural networks
Ilya Sutskever, James Martens, and Geoffrey E. Hinton · 2011
Earlier work this paper cites.
Supervised Sequence Labelling with Recurrent Neural Networks
Alex Graves · 2012
Earlier work this paper cites.
ImageNet classification with deep convolutional neural networks
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton · 2012
Earlier work this paper cites.
Generating sequences with recurrent neural networks
Original
Alex Graves · 2013
Earlier work this paper cites.
On the properties of neural machine translation: Encoder-decoder approaches
Original
Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio · 2014
Earlier work this paper cites.
Sequence to sequence learning with neural networks
Ilya Sutskever, Oriol Vinyals, and Quoc V Le · 2014
Earlier work this paper cites.
Neural machine translation by jointly learning to align and translate
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio · 2015
Earlier work this paper cites.
Long-term recurrent convolutional networks for visual recognition and description
Jeff Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Trevor Darrell, and Kate Saenko · 2015
Earlier work this paper cites.
An empirical exploration of recurrent network architectures
Rafal Jozefowicz, Wojciech Zaremba, and Ilya Sutskever · 2015
Earlier work this paper cites.
Deep visual-semantic alignments for generating image descriptions
Andrej Karpathy and Fei-Fei Li · 2015
Earlier work this paper cites.

Similar

Deeply-supervised nets
Chen-Yu Lee, Saining Xie, Patrick Gallagher, Zhengyou Zhang, and Zhuowen Tu · 2015
Cited alongside, same era.
Convolutional, long short-term memory, fully connected deep neural networks
Tara N. Sainath, Oriol Vinyals, Andrew W. Senior, and Hasim Sak · 2015
Cited alongside, same era.
Convolutional LSTM network: A machine learning approach for precipitation nowcasting
Xingjian Shi, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-Kin Wong, and Wang-chun Woo · 2015
Cited alongside, same era.
Sequence to sequence – video to text
Subhashini Venugopalan, Marcus Rohrbach, Jeffrey Donahue, Raymond J. Mooney, Trevor Darrell, and Kate Saenko · 2015
Cited alongside, same era.
Show and tell: A neural image caption generator
Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan · 2015
Cited alongside, same era.
Holistically-nested edge detection
Saining Xie and Zhuowen Tu · 2015
Cited alongside, same era.
A theoretically grounded application of dropout in recurrent neural networks
Yarin Gal and Zoubin Ghahramani · 2016
Cited alongside, same era.
Neural machine translation in linear time
Original
Nal Kalchbrenner, Lasse Espeholt, Karen Simonyan, Aäron van den Oord, Alex Graves, and Koray Kavukcuoglu · 2016
Cited alongside, same era.
Gated word-character recurrent language model
Original
Yasumasa Miyamoto and Kyunghyun Cho · 2016
Cited alongside, same era.
Weight normalization: A simple reparameterization to accelerate training of deep neural networks
Tim Salimans and Diederik P Kingma · 2016
Cited alongside, same era.
WaveNet: A generative model for raw audio
Original
Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew W. Senior, and Koray Kavukcuoglu · 2016
Cited alongside, same era.
Multi-scale context aggregation by dilated convolutions
Fisher Yu and Vladlen Koltun · 2016
Cited alongside, same era.

Then

Pointer sentinel mixture models
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher · 2017
Later among the works it cites.
Fast-slow recurrent neural networks
Asier Mujika, Florian Meier, and Angelika Steger · 2017
Later among the works it cites.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin · 2017
Later among the works it cites.
A primal dual network for low-level vision problems
Christoph Vogel and Thomas Pock · 2017
Later among the works it cites.
Recurrent highway networks
Julian Georg Zilly, Rupesh Kumar Srivastava, Jan Koutník, and Jürgen Schmidhuber · 2017
Later among the works it cites.
Neural architecture search with reinforcement learning
Barret Zoph and Quoc V Le · 2017
Later among the works it cites.
An empirical evaluation of generic convolutional and recurrent networks for sequence modeling
Original
Shaojie Bai, J Zico Kolter, and Vladlen Koltun · 2018
Closest in time.
The best of both worlds: Combining recent advances in neural machine translation
Mia Xu Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Foster, Llion Jones, Niki Parmar, Noam Shazeer, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Mike Schuster, Zhifeng Chen, Yonghui Wu, and Macduff Hughes · 2018
Closest in time.
Sharp nearby, fuzzy far away: How neural language models use context
Urvashi Khandelwal, He He, Peng Qi, and Dan Jurafsky · 2018
Closest in time.
Independently recurrent neural network (IndRNN): Building a longer and deeper RNN
Shuai Li, Wanqing Li, Chris Cook, Ce Zhu, and Yanbo Gao · 2018
Closest in time.
DARTS: Differentiable architecture search
Original
Hanxiao Liu, Karen Simonyan, and Yiming Yang · 2018
Closest in time.
On the state of the art of evaluation in neural language models
Gábor Melis, Chris Dyer, and Phil Blunsom · 2018
Closest in time.
When recurrent models don’t need to be recurrent
Original
John Miller and Moritz Hardt · 2018
Closest in time.
Efficient neural architecture search via parameters sharing
Hieu Pham, Melody Y Guan, Barret Zoph, Quoc V Le, and Jeff Dean · 2018
Closest in time.
Relational recurrent neural networks
Adam Santoro, Ryan Faulkner, David Raposo, Jack Rae, Mike Chrzanowski, Theophane Weber, Daan Wierstra, Oriol Vinyals, Razvan Pascanu, and Timothy Lillicrap · 2018
Closest in time.
Learning longer-term dependencies in RNNs with auxiliary losses
Trieu H Trinh, Andrew M Dai, Thang Luong, and Quoc V Le · 2018
Closest in time.
Breaking the softmax bottleneck: A high-rank RNN language model
Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, and William W. Cohen · 2018
Closest in time.
Transformer-XL: Language modeling with longer-term dependency
Original
Zihang Dai, Zhilin Yang, Yiming Yang, William W Cohen, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…