Fetching the paper…

Reading the bibliography…

2017

Unbiasing Truncated Backpropagation Through Time

Tallec, Corentin, Ollivier, Yann

Original
PDF
Post on X

Understand

Truncated Backpropagation Through Time (truncated BPTT) is a widespread method for learning recurrent computational graphs.

Truncated BPTT keeps the computational benefits of Backpropagation Through Time (BPTT) while relieving the need for a complete backtrack through the whole data sequence at every step.
However, truncation favors short-term dependencies: the gradient estimate of truncated BPTT is biased, so that it does not benefit from the convergence guarantees from stochastic gradient theory.
We introduce Anticipated Reweighted Truncated Backpropagation (ARTBP), an algorithm that keeps the computational benefits of truncated BPTT, while providing unbiasedness.

Unbiasing Truncated Backpropagation Through Time · Around

Built on

Backpropagation through time: what does it do and how to do it
P. Werbos · 1990
Earlier work this paper cites.
Building a large annotated corpus of English: The Penn Treebank
Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz · 1993
Earlier work this paper cites.
Long short-term memory
Sepp Hochreiter and Jürgen Schmidhuber · 1997
Earlier work this paper cites.
Learning to forget: Continual prediction with LSTM
Felix A. Gers, Jürgen A. Schmidhuber, and Fred A. Cummins · 2000
Earlier work this paper cites.
A tutorial on training recurrent neural networks, covering BPPT, RTRL, EKF and the "echo state network" approach
Herbert Jaeger · 2005
Earlier work this paper cites.

Similar

Subword language modeling with neural networks
Tomás̆ Mikolov, Ilya Sutskever, Anoop Deoras, Le Hai-Son, Stefan Kombrink, and Jan C̆ernocký · 2012
Cited alongside, same era.
Generating sequences with recurrent neural networks
Original
Alex Graves · 2013
Cited alongside, same era.
Training Recurrent Neural Networks
Ilya Sutskever · 2013
Cited alongside, same era.
Adam: A method for stochastic optimization
Original
Diederik P. Kingma and Jimmy Ba · 2014
Cited alongside, same era.

Then

Training recurrent networks online without backtracking
Original
Yann Ollivier, Corentin Tallec, and Guillaume Charpiat · 2015
Later among the works it cites.
Recurrent batch normalization
Original
Tim Cooijmans, Nicolas Ballas, César Laurent, and Aaron C. Courville · 2016
Later among the works it cites.
Memory-efficient backpropagation through time
Audrunas Gruslys, Rémi Munos, Ivo Danihelka, Marc Lanctot, and Alex Graves · 2016
Later among the works it cites.
Unbiased online recurrent optimization
Original
Corentin Tallec and Yann Ollivier · 2017
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…