Fetching the paper…

Reading the bibliography…

2017

On the Long-Term Memory of Deep Recurrent Networks

Levine, Yoav, Sharir, Or, Ziv, Alon et al.

Original
PDF
Post on X

Understand

A key attribute that drives the unprecedented success of modern Recurrent Neural Networks (RNNs) on learning tasks which involve sequential data, is their ability to model intricate long-term temporal dependencies.

However, a well established measure of RNNs long-term memory capacity is lacking, and thus formal understanding of the effect of depth on their ability to correlate data throughout time is limited.
Specifically, existing depth efficiency results on convolutional networks do not suffice in order to account for the success of deep RNNs on data of varying lengths.
In order to address this, we introduce a measure of the network's ability to support information flow across time, referred to as the Start-End separation rank, which reflects the distance of the function realized by the recurrent network from modeling no dependency between the beginning and end of the input sequence.

On the Long-Term Memory of Deep Recurrent Networks · Around

Built on

Hardy, G. H., Littlewood, J. E. &
1952
Earlier work this paper cites.
Schmidhuber, J. H
1992
Earlier work this paper cites.
Hochreiter, S. &
1997
Earlier work this paper cites.
LeCun, Y., Cortes, C. &
1998
Earlier work this paper cites.
Gers, F. A. &
2000
Earlier work this paper cites.
Beylkin, G. &
2002
Earlier work this paper cites.
Harrison, R. J., Fann, G. I., Yanai, T. &
2003
Earlier work this paper cites.
Hackbusch, W
2006
Earlier work this paper cites.
Beylkin, G., Garcke, J. &
2009
Earlier work this paper cites.
Graves, A., Liwicki, M., Fernández, S., Bertolami, R., Bunke, H. &
2009
Earlier work this paper cites.
Cireşan, D. C., Meier, U., Gambardella, L. M. &
2010
Earlier work this paper cites.
Delalleau, O. &
2011
Earlier work this paper cites.
Martens, J. &
2011
Earlier work this paper cites.
Oseledets, I. V
2011
Earlier work this paper cites.

Similar

Poon, H. &
2011
Cited alongside, same era.
Sutskever, I., Martens, J. &
2011
Cited alongside, same era.
Amini, A., Karbasi, A. &
2012
Cited alongside, same era.
Mohamed, A.-r., Dahl, G. E. &
2012
Cited alongside, same era.
Tieleman, T. &
2012
Cited alongside, same era.
Graves, A
Original
2013
Cited alongside, same era.
Graves, A., Mohamed, A.-r. &
2013
Cited alongside, same era.

Then

Cohen, N., Sharir, O. &
2016
Later among the works it cites.
Cohen, N. &
2016
Later among the works it cites.
Eldan, R. &
2016
Later among the works it cites.
Jing, L., Shen, Y., Dubček, T., Peurifoy, J., Skirlo, S., LeCun, Y., Tegmark, M. &
Original
2016
Later among the works it cites.
Sharir, O., Tamari, R., Cohen, N. &
2016
Later among the works it cites.
Stoudenmire, E. &
2016
Later among the works it cites.
Wisdom, S., Powers, T., Hershey, J., Le Roux, J. &
2016
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Hermans, M. &

2013

Cited alongside, same era.

Pascanu, R., Mikolov, T. &

2013

Cited alongside, same era.

Bahdanau, D., Cho, K. &

2014

Cited alongside, same era.

Cho, K., Van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H. &

2014

Cited alongside, same era.

2015

Cited alongside, same era.

Wu, Y., Zhang, S., Zhang, Y., Bengio, Y. &

2016

Later among the works it cites.

Zhang, S., Wu, Y., Che, T., Lin, Z., Memisevic, R., Salakhutdinov, R. R. &

2016

Later among the works it cites.

Cohen, N., Tamari, R. &

2017

Closest in time.

Khrulkov, V., Novikov, A. &

2018

Closest in time.

Levine, Y., Yakira, D., Cohen, N. &

2018

Closest in time.

Sharir, O. &

2018

Closest in time.