Fetching the paper…

Reading the bibliography…

2017

On Quadratic Penalties in Elastic Weight Consolidation

Huszár, Ferenc

Original
PDF
Post on X

Understand

Elastic weight consolidation (EWC, Kirkpatrick et al, 2017) is a novel algorithm designed to safeguard against catastrophic forgetting in neural networks.

EWC can be seen as an approximation to Laplace propagation (Eskin et al, 2004), and this view is consistent with the motivation given by Kirkpatrick et al (2017).
In this note, I present an extended derivation that covers the case when there are more than two tasks.
I show that the quadratic penalties in EWC are inconsistent with this derivation and might lead to double-counting data from earlier tasks.

On Quadratic Penalties in Elastic Weight Consolidation · Around

Built on

On convergence of posterior distributions
Subhashis Ghosal, Jayanta K Ghosh, and Tapas Samanta · 1995
Earlier work this paper cites.
A Bayesian approach to on-line learning
Original
Manfred Opper and Ole Winther · 1998
Earlier work this paper cites.
Expectation propagation for approximate bayesian inference
Thomas P Minka · 2001
Earlier work this paper cites.

Similar

Information theory, inference and learning algorithms
David JC MacKay · 2003
Cited alongside, same era.
Laplace propagation
Eleazar Eskin, Alex J Smola, and SVN Vishwanathan · 2004
Cited alongside, same era.
New insights and perspectives on the natural gradient method
Original
James Martens · 2014
Cited alongside, same era.

Then

Deep Learning
Ian Goodfellow, Yoshua Bengio, and Aaron Courville · 2016
Later among the works it cites.
Overcoming catastrophic forgetting in neural networks
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell · 2017
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…