2020

Dropout as an Implicit Gating Mechanism For Continual Learning

Mirzadeh, Seyed-Iman, Farajtabar, Mehrdad, Ghasemzadeh, Hassan

Understand

In recent years, neural networks have demonstrated an outstanding ability to achieve complex learning tasks across various domains.

However, they suffer from the "catastrophic forgetting" problem when they face a sequence of learning tasks, where they forget the old ones as they learn new tasks.
This problem is also highly related to the "stability-plasticity dilemma".
The more plastic the network, the easier it can learn new tasks, but the faster it also forgets previous ones.

Catastrophic interference in connectionist networks: The sequential learning problem
Michael McCloskey and Neal J. Cohen · 1989
Earlier work this paper cites.
Improving neural networks by preventing co-adaptation of feature detectors
Original
Geoffrey E. Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov · 2012
Earlier work this paper cites.
Understanding dropout
Pierre Baldi and Peter J Sadowski · 2013
Earlier work this paper cites.
An empirical investigation of catastrophic forgeting in gradient-based neural networks
Original
Ian J. Goodfellow, Mehdi Mirza, Xia Da, Aaron C. Courville, and Yoshua Bengio · 2013
Earlier work this paper cites.
The stability-plasticity dilemma: investigating the continuum from catastrophic forgetting to age-limited learning effects
Martial Mermillod, Aurélia Bugaiska, and Patrick Bonin · 2013
Earlier work this paper cites.
Improving neural networks with dropout
Nitish Srivastava · 2013
Earlier work this paper cites.
The log-dynamic brain: how skewed distributions affect network operations
György Buzsáki and Kenji Mizuseki · 2014
Earlier work this paper cites.
Distilling the knowledge in a neural network
Original
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean · 2015
Earlier work this paper cites.
Surprising properties of dropout in deep networks
David P. Helmbold and Philip M. Long · 2016
Earlier work this paper cites.

icarl: Incremental classifier and representation learning
Sylvestre-Alvise Rebuffi, Alexander I Kolesnikov, Georg Sperl, and Christoph H. Lampert · 2016
Cited alongside, same era.
Overcoming catastrophic forgetting in neural networks
James N Kirkpatrick, Razvan Pascanu, Neil C. Rabinowitz, Joel Veness, and et. al · 2017
Cited alongside, same era.
Packnet: Adding multiple tasks to a single network by iterative pruning
Arun Mallya and Svetlana Lazebnik · 2017
Cited alongside, same era.
Continual learning through synaptic intelligence
Friedemann Zenke, Ben Poole, and Surya Ganguli · 2017
Cited alongside, same era.
Riemannian walk for incremental learning: Understanding forgetting and intransigence
Arslan Chaudhry, Puneet Kumar Dokania, Thalaiyasingam Ajanthan, and Philip H. S. Torr · 2018
Cited alongside, same era.
Efficient lifelong learning with a-gem
Original
Arslan Chaudhry, Marc’Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny · 2018
Cited alongside, same era.
Learning without forgetting
Zhizhong Li and Derek Hoiem · 2018
Cited alongside, same era.
Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilization
Nicolas Y. Masse, Gregory D. Grant, and David J. Freedman · 2018
Cited alongside, same era.
Using hindsight to anchor past knowledge in continual learning
Original
Arslan Chaudhry, Albert Gordo, Puneet Kumar Dokania, Philip H. S. Torr, and David Lopez-Paz · 2019
Cited alongside, same era.

Orthogonal gradient descent for continual learning
Original
Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li · 2019
Later among the works it cites.
Continual learning: A comparative study on how to defy forgetting in classification tasks
Original
Matthias Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ale · 2019
Later among the works it cites.
Overcoming catastrophic forgetting with unlabeled data in the wild
Kibok Lee, Kimin Lee, Jinwoo Shin, and Honglak Lee · 2019
Later among the works it cites.
Improved knowledge distillation via teacher assistant: Bridging the gap between student and teacher
Original
Seyed-Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, and Hassan Ghasemzadeh · 2019
Later among the works it cites.
Pytorch: An imperative style, high-performance deep learning library
Adam Paszke, Sam Gross, Francisco Massa, and et. al · 2019
Later among the works it cites.
Actilabel: A combinatorial transfer learning framework for activity recognition
Original
Parastoo Alinia, Seyed-Iman Mirzadeh, and Hassan Ghasemzadeh · 2020
Closest in time.
Self-distillation amplifies regularization in hilbert space
Original
Hossein Mobahi, Mehrdad Farajtabar, and Peter L. Bartlett · 2020
Closest in time.
The implicit and explicit regularization effects of dropout
Original
Colin Wei, Sham M. Kakade, and Tengyu Ma · 2020
Closest in time.

alphaXiv searches the wider corpus for related work and actual follow-ups.

alphaXiv is searching for related work…

Reading the bibliography…