2016

Architectural Complexity Measures of Recurrent Neural Networks

Zhang, Saizheng, Wu, Yuhuai, Che, Tong et al.

Understand

In this paper, we systematically analyze the connecting architectures of recurrent neural networks (RNNs).

Our main contribution is twofold: first, we present a rigorous graph-theoretic framework describing the connecting architectures of RNNs in general.
Second, we propose three architecture complexity measures of RNNs: (a) the recurrent depth, which captures the RNN's over-time nonlinear complexity, (b) the feedforward depth, which captures the local input-output nonlinearity (similar to the "depth" in feedforward neural networks (FNNs)), and (c) the recurrent skip coefficient which captures how rapidly the information propagates over time.
We rigorously prove each measure's existence and computability.

Built on

Untersuchungen zu dynamischen neuronalen netzen
Sepp Hochreiter · 1991
Earlier work this paper cites.
Untersuchungen zu dynamischen neuronalen netzen
Sepp Hochreiter · 1991
Earlier work this paper cites.
Learning complex, extended sequences using the principle of history compression
Jürgen Schmidhuber · 1992
Earlier work this paper cites.
Learning complex, extended sequences using the principle of history compression
Jürgen Schmidhuber · 1992
Earlier work this paper cites.
Learning long-term dependencies with gradient descent is difficult
Yoshua Bengio, Patrice Simard, and Paolo Frasconi · 1994
Earlier work this paper cites.
Learning long-term dependencies with gradient descent is difficult
Yoshua Bengio, Patrice Simard, and Paolo Frasconi · 1994
Earlier work this paper cites.
Hierarchical recurrent neural networks for long-term dependencies
Salah El Hihi and Yoshua Bengio · 1996
Earlier work this paper cites.
Learning long-term dependencies is not as difficult with NARX recurrent neural networks
T. Lin, B. G. Horne, P. Tino, and C. L. Giles · 1996
Earlier work this paper cites.
Hierarchical recurrent neural networks for long-term dependencies
Salah El Hihi and Yoshua Bengio · 1996
Earlier work this paper cites.
Learning long-term dependencies is not as difficult with NARX recurrent neural networks
T. Lin, B. G. Horne, P. Tino, and C. L. Giles · 1996
Earlier work this paper cites.
Long short-term memory
Sepp Hochreiter and Jürgen Schmidhuber · 1997
Earlier work this paper cites.
Long short-term memory
Sepp Hochreiter and Jürgen Schmidhuber · 1997
Earlier work this paper cites.
Multi-dimensional recurrent neural networks
Alex Graves, Santiago Fernández, and Jürgen Schmidhuber · 2007
Earlier work this paper cites.
Multi-dimensional recurrent neural networks
Alex Graves, Santiago Fernández, and Jürgen Schmidhuber · 2007
Earlier work this paper cites.
Temporal-kernel recurrent neural networks
Ilya Sutskever and Geoffrey Hinton · 2010
Earlier work this paper cites.
Temporal-kernel recurrent neural networks
Ilya Sutskever and Geoffrey Hinton · 2010
Earlier work this paper cites.
Learning recurrent neural networks with hessian-free optimization
James Martens and Ilya Sutskever · 2011
Earlier work this paper cites.
Learning recurrent neural networks with hessian-free optimization
James Martens and Ilya Sutskever · 2011
Earlier work this paper cites.

Similar

Subword language modeling with neural networks
Tomáš Mikolov, Ilya Sutskever, Anoop Deoras, Hai-Son Le, and Stefan Kombrink · 2012
Cited alongside, same era.
Understanding the exploding gradient problem
Original
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio · 2012
Cited alongside, same era.
Deep learning made easier by linear transformations in perceptrons
Tapani Raiko, Harri Valpola, and Yann LeCun · 2012
Cited alongside, same era.
Subword language modeling with neural networks
Tomáš Mikolov, Ilya Sutskever, Anoop Deoras, Hai-Son Le, and Stefan Kombrink · 2012
Cited alongside, same era.
Understanding the exploding gradient problem
Original
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio · 2012
Cited alongside, same era.
Deep learning made easier by linear transformations in perceptrons
Tapani Raiko, Harri Valpola, and Yann LeCun · 2012
Cited alongside, same era.
Generating sequences with recurrent neural networks
Original
Alex Graves · 2013
Cited alongside, same era.
Training and analysing deep recurrent neural networks
Michiel Hermans and Benjamin Schrauwen · 2013
Cited alongside, same era.
How to construct deep recurrent neural networks
Original
Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio · 2013
Cited alongside, same era.
On the difficulty of training recurrent neural networks
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio · 2013
Cited alongside, same era.
Generating sequences with recurrent neural networks
Original
Alex Graves · 2013
Cited alongside, same era.
Training and analysing deep recurrent neural networks
Michiel Hermans and Benjamin Schrauwen · 2013
Cited alongside, same era.

Then

Sequence to sequence learning with neural networks
Ilya Sutskever, Oriol Vinyals, and Quoc VV Le · 2014
Later among the works it cites.
Neural machine translation by jointly learning to align and translate
Original
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio · 2014
Later among the works it cites.
Learning phrase representations using rnn encoder-decoder for statistical machine translation
Original
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio · 2014
Later among the works it cites.
Adam: A method for stochastic optimization
Original
Diederik Kingma and Jimmy Ba · 2014
Later among the works it cites.
A clockwork rnn
Jan Koutnik, Klaus Greff, Faustino Gomez, and Juergen Schmidhuber · 2014
Later among the works it cites.
Sequence to sequence learning with neural networks
Ilya Sutskever, Oriol Vinyals, and Quoc VV Le · 2014
Later among the works it cites.
Unitary evolution recurrent neural networks
Original
Martin Arjovsky, Amar Shah, and Yoshua Bengio · 2015
Later among the works it cites.
Lstm: A search space odyssey
Original
Klaus Greff, Rupesh Kumar Srivastava, Jan Koutník, Bas R Steunebrink, and Jürgen Schmidhuber · 2015
Later among the works it cites.
An empirical exploration of recurrent network architectures
Rafal Jozefowicz, Wojciech Zaremba, and Ilya Sutskever · 2015
Later among the works it cites.
Skip-thought vectors
Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S. Zemel, Antonio Torralba, Raquel Urtasun, and Sanja Fidler · 2015
Later among the works it cites.
A simple way to initialize recurrent networks of rectified linear units
Original
Quoc V Le, Navdeep Jaitly, and Geoffrey E Hinton · 2015
Later among the works it cites.
Unsupervised learning of video representations using LSTMs
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhutdinov · 2015
Later among the works it cites.
Unitary evolution recurrent neural networks
Original
Martin Arjovsky, Amar Shah, and Yoshua Bengio · 2015
Later among the works it cites.
Lstm: A search space odyssey
Original
Klaus Greff, Rupesh Kumar Srivastava, Jan Koutník, Bas R Steunebrink, and Jürgen Schmidhuber · 2015
Later among the works it cites.
An empirical exploration of recurrent network architectures
Rafal Jozefowicz, Wojciech Zaremba, and Ilya Sutskever · 2015
Later among the works it cites.
Skip-thought vectors
Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S. Zemel, Antonio Torralba, Raquel Urtasun, and Sanja Fidler · 2015
Later among the works it cites.
A simple way to initialize recurrent networks of rectified linear units
Original
Quoc V Le, Navdeep Jaitly, and Geoffrey E Hinton · 2015
Later among the works it cites.
Unsupervised learning of video representations using LSTMs
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhutdinov · 2015
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…