Fetching the paper…
Reading the bibliography…
A recent strategy to circumvent the exploding and vanishing gradient problem in RNNs, and to allow the stable propagation of signals over long time scales, is to constrain recurrent connectivity matrices to be orthogonal or unitary.
Learning long-term dependencies with gradient descent is difficult
Y Bengio, P Simard, and P Frasconi · 1994
Earlier work this paper cites.
Long short-term memory
Sepp Hochreiter and Jürgen Schmidhuber · 1997
Earlier work this paper cites.
LAPACK Users’ Guide (Third Ed.)
E. Anderson, Z. Bai, C. Bischof, L. S. Blackford, J. Demmel, Jack J. Dongarra, J. Du Croz, S. Hammarling, A. Greenbaum, A. McKenney, and D. Sorensen · 1999
Earlier work this paper cites.
Linear Algebra and Its Applications
Peter D. Lax · 2007
Earlier work this paper cites.
Memory traces in dynamical systems
Surya Ganguli, Dongsung Huh, and Haim Sompolinsky · 2008
Earlier work this paper cites.
Memory without Feedback in a Neural Network
Mark S Goldman · 2009
Earlier work this paper cites.
Non-normal amplification in random balanced neuronal networks
Guillaume Hennequin, Tim P. Vogels, and Wulfram Gerstner · 2012
Earlier work this paper cites.
On the difficulty of training recurrent neural networks
R Pascanu, T Mikolov, and Y Bengio · 2013
Earlier work this paper cites.
Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
Andrew M Saxe, James L McClelland, and Surya Ganguli · 2014
Earlier work this paper cites.
A framework of constraint preserving update schemes for optimization on Stiefel manifold
Bo Jiang and Yu-Hong Dai · 2015
Earlier work this paper cites.
A simple way to initialize recurrent networks of rectified linear units
Quoc V Le, Navdeep Jaitly, and Geoffrey E Hinton · 2015
Earlier work this paper cites.
Deep fried convnets
Z. Yang, M. Moczulski, M. Denil, N. d. Freitas, A. Smola, L. Song, and Z. Wang · 2015
Earlier work this paper cites.
Unitary evolution recurrent neural networks
Martin Arjovsky, Amar Shah, and Yoshua Bengio · 2016
Cited alongside, same era.
Recurrent orthogonal networks and long-memory tasks
Mikael Henaff, Arthur Szlam, and Yann LeCun · 2016
Cited alongside, same era.
Full-Capacity Unitary Recurrent Neural Networks
Scott Wisdom, Thomas Powers, John R Hershey, Jonathan Le Roux, and Les Atlas · 2016
Cited alongside, same era.
Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNNs
Li Jing, Yichen Shen, John Peurifoy, Scott Skirlo, Yann LeCun, and Max Tegmark · 2017
Cited alongside, same era.
Building a large annotated corpus of english: The penn treebank
Mitchell P. Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini · 2017
Cited alongside, same era.
Efficient Orthogonal Parametrisation of Recurrent Neural Networks Using Householder Reflections
Zakaria Mhammedi, Andrew Hellicar, Ashfaqur Rahman, and James Bailey · 2017
Independently recurrent neural network (indrnn): Building a longer and deeper rnn
Shuai Li, Wanqing Li, Chris Cook, Ce Zhu, and Yanbo Gao · 2018
Later among the works it cites.
Can recurrent neural networks warp time?
Corentin Tallec and Yann Ollivier · 2018
Later among the works it cites.
Stabilizing Gradients for Deep Neural Networks via Efficient SVD Parameterization
Jiong Zhang, Qi Lei, and Inderjit Dhillon · 2018
Later among the works it cites.
h-detach: Modifying the LSTM Gradient Towards Better Optimization
Devansh Arpit, Bhargav Kanuparthi, Giancarlo Kerg, Nan Rosemary Ke, Ioannis Mitliagkas, and Yoshua Bengio · 2019
Closest in time.
Towards Non-saturating Recurrent Units for Modelling Long-term Dependencies
Sarath Chandar, Chinnadhurai Sankar, Eugene Vorontsov, Samira Ebrahimi Kahou, and Yoshua Bengio · 2019
Closest in time.
AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networks
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Cited alongside, same era.
Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice
Jeffrey Pennington, Samuel S Schoenholz, and Surya Ganguli · 2017
Cited alongside, same era.
On the expressive power of deep neural networks
Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, and Jascha Sohl Dickstein · 2017
Cited alongside, same era.
On orthogonality and learning rnn with long term dependencies
Eugene Vorontsov, Chiheb Trabelsi, Samuel Kadoury, and Christopher Pal · 2017
Cited alongside, same era.
Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks
Minmin Chen, Jeffrey Pennington, and Samuel S Schoenholz · 2018
Cited alongside, same era.
Orthogonal Recurrent Neural Networks with Scaled Cayley Transform
Kyle Helfrich, Devin Willmott, and Qiang Ye · 2018
Cited alongside, same era.
Bo Chang, Minmin Chen, Eldad Haber, and Ed H Chi · 2019
Closest in time.
Gated orthogonal recurrent units: On learning to forget
L Jing, C Gulcehre, J Peurifoy, Y Shen, M Tegmark Neural, and 2019 · 2019
Closest in time.
Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group
Mario Lezcano-Casado and David Martínez-Rubio · 2019
Closest in time.
Complex Unitary Recurrent Neural Networks using Scaled Cayley Transform
Kehelwala D G Maduranga, Kyle E Helfrich, and Qiang Ye · 2019
Closest in time.
Improved memory in recurrent neural networks with sequential non-normal dynamics
A Emin Orhan and Xaq Pitkow · 2019
Closest in time.
Quaternion Recurrent Neural Networks
Titouan Parcollet, Mirco Ravanelli, Mohamed Morchid, Georges Linarès, Chiheb Trabelsi, Renato De Mori, and Yoshua Bengio · 2019
Closest in time.