Fetching the paper…

Reading the bibliography…

2016

Surprisal-Driven Zoneout

Rocki, Kamil, Kornuta, Tomasz, Maharaj, Tegan

Original
PDF
Post on X

Understand

We propose a novel method of regularization for recurrent neural networks called suprisal-driven zoneout.

In this method, states zoneout (maintain their previous value rather than updating), when the suprisal (discrepancy between the last state's prediction and target) is small.
Thus regularization is adaptive and input-driven on a per-neuron basis.
We demonstrate the effectiveness of this idea by achieving state-of-the-art bits per character of 1.31 on the Hutter Prize Wikipedia dataset, significantly reducing the gap to the best known highly-engineered compression methods.

Surprisal-Driven Zoneout · Around

Built on

Possible principles underlying the transformations of sensory messages
H. B. Barlow · 1961
Earlier work this paper cites.
A formal theory of inductive inference. part i
R. J. Solomonoff · 1964
Earlier work this paper cites.
Long short-term memory
S. Hochreiter and J. Schmidhuber · 1997
Earlier work this paper cites.
Understanding the difficulty of training deep feedforward neural networks
X. Glorot and Y. Bengio · 2010
Earlier work this paper cites.
Generating text with recurrent neural networks
I. Sutskever, J. Martens, and G. Hinton · 2011
Earlier work this paper cites.

Similar

Gated feedback recurrent neural networks
Original
J. Chung, Ç. Gülçehre, K. Cho, and Y. Bengio · 2015
Cited alongside, same era.
Grid long short-term memory
Original
N. Kalchbrenner, I. Danihelka, and A. Graves · 2015
Cited alongside, same era.
Layer normalization
Original
J. L. Ba, J. R. Kiros, and G. E. Hinton · 2016
Cited alongside, same era.
Hierarchical multiscale recurrent neural networks
Original
J. Chung, S. Ahn, and Y. Bengio · 2016
Cited alongside, same era.
Recurrent memory array structures
Original
K. Rocki
Cited in the paper.
Surprisal-driven feedback in recurrent networks
Original
K. M. Rocki
Cited in the paper.

Then

Hypernetworks
Original
D. Ha, A. Dai, and Q. V. Le · 2016
Closest in time.
Zoneout: Regularizing rnns by randomly preserving hidden activations
Original
D. Krueger, T. Maharaj, J. Kramár, M. Pezeshki, N. Ballas, N. R. Ke, A. Goyal, Y. Bengio, H. Larochelle, A. C. Courville, and C. Pal · 2016
Closest in time.
On multiplicative integration with recurrent neural networks
Original
Y. Wu, S. Zhang, Y. Zhang, Y. Bengio, and R. Salakhutdinov · 2016
Closest in time.
Recurrent highway networks
Original
J. G. Zilly, R. K. Srivastava, J. Koutník, and J. Schmidhuber · 2016
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…