2017

FearNet: Brain-Inspired Model for Incremental Learning

Kemker, Ronald, Kanan, Christopher

Understand

Incremental class learning involves sequentially learning classes in bursts of examples from the same class.

This violates the assumptions that underlie methods for training standard deep neural networks, and will cause them to suffer from catastrophic forgetting.
Arguably, the best method for incremental class learning is iCaRL, but it requires storing training examples for each class, making it challenging to scale.
Here, we propose FearNet for incremental class learning.

Built on

Autoradiographic investigation of cell proliferation in the brains of rats and cats
Joseph Altman · 1963
Earlier work this paper cites.
Is there “catastrophic interference” in connectionist networks
P Hetherington and Mark S Seidenberg · 1989
Earlier work this paper cites.
Catastrophic interference in connectionist networks: The sequential learning problem
Michael McCloskey and Neal J Cohen · 1989
Earlier work this paper cites.
Probabilistic neural networks
Donald F Specht · 1990
Earlier work this paper cites.
The Stability/Plasticity Dilemma in Self-organising Neural Networks
AJ Gillies · 1991
Earlier work this paper cites.
Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory
James L McClelland, Bruce L McNaughton, and Randall C O’reilly · 1995
Earlier work this paper cites.
Catastrophic forgetting, rehearsal and pseudorehearsal
Anthony Robins · 1995
Earlier work this paper cites.
Avoiding catastrophic forgetting by coupling two reverberating neural networks
Bernard Ans and Stéphane Rousset · 1997
Earlier work this paper cites.
Pseudo-recurrent connectionist networks: An approach to the ‘sensitivity-stability’ dilemma
Robert M French · 1997
Earlier work this paper cites.
Neurogenesis in the adult human hippocampus
Peter S Eriksson, Ekaterina Perfilieva, Thomas Björk-Eriksson, Ann-Marie Alborn, Claes Nordborg, Daniel A Peterson, and Fred H Gage · 1998
Earlier work this paper cites.
Time-dependent reorganization of brain circuitry underlying long-term memory storage
Bruno Bontempi, Catherine Laurent-Demir, Claude Destrade, and Robert Jaffard · 1999
Earlier work this paper cites.
Catastrophic forgetting in connectionist networks
Robert M French · 1999
Earlier work this paper cites.
A fast algorithm for the minimum covariance determinant estimator
Peter J Rousseeuw and Katrien Van Driessen · 1999
Earlier work this paper cites.
Estimating the support of a high-dimensional distribution
Bernhard Schölkopf, John C. Platt, John C. Shawe-Taylor, Alex J. Smola, and Robert C. Williamson · 2001
Earlier work this paper cites.

Similar

Adult neurogenesis and neural stem cells of the central nervous system in mammals
Philippe Taupin and Fred H Gage · 2002
Cited alongside, same era.
The involvement of the anterior cingulate cortex in remote contextual fear memory
Paul W Frankland, Bruno Bontempi, Lynn E Talton, Leszek Kaczmarek, and Alcino J Silva · 2004
Cited alongside, same era.
Memory retention–the synaptic stability versus plasticity dilemma
Wickliffe C Abraham and Anthony Robins · 2005
Cited alongside, same era.
Declarative memory consolidation in humans: a prospective functional magnetic resonance imaging study
A Takashima, Karl Magnus Petersson, F Rutters, I Tendolkar, O Jensen, MJ Zwarts, BL McNaughton, and G Fernandez · 2006
Cited alongside, same era.
Sleep transforms the cerebral trace of declarative memories
Steffen Gais, Geneviève Albouy, Mélanie Boly, Thien Thanh Dang-Vu, Annabelle Darsaud, Martin Desseilles, Géraldine Rauchs, Manuel Schabus, Virginie Sterpenich, Gilles Vandewalle, et al · 2007
Cited alongside, same era.
A deficit in the ability to form new human memories without sleep
Seung-Schik Yoo, Peter T Hu, Ninad Gujar, Ferenc A Jolesz, and Matthew P Walker · 2007
Cited alongside, same era.
Isolation forest
Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou · 2008
Cited alongside, same era.
New neurons and new memories: how does adult hippocampal neurogenesis affect learning and memory?
Wei Deng, James B Aimone, and Fred H Gage · 2010
Cited alongside, same era.
Understanding the difficulty of training deep feedforward neural networks
Xavier Glorot and Yoshua Bengio · 2010
Cited alongside, same era.
Caltech-UCSD Birds 200
P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona · 2010
Cited alongside, same era.
The role of medial prefrontal cortex in memory and decision making
David R Euston, Aaron J Gruber, and Bruce L McNaughton · 2012
Cited alongside, same era.
Imagenet classification with deep convolutional neural networks
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton · 2012
Cited alongside, same era.

Then

About sleeps role in memory
Björn Rasch and Jan Born · 2013
Later among the works it cites.
Semi-supervised learning with ladder networks
Antti Rasmus, Mathias Berglund, Mikko Honkala, Harri Valpola, and Tapani Raiko · 2015
Later among the works it cites.
Youtube-8m: A large-scale video classification benchmark
Original
Sami Abu-El-Haija, Nisarg Kothari, Joonseok Lee, et al · 2016
Later among the works it cites.
Fast and accurate deep network learning by exponential linear units (elus)
Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter · 2016
Later among the works it cites.
A bio-inspired incremental learning architecture for applied perceptual problems
Alexander Gepperth and Cem Karaoguz · 2016
Later among the works it cites.
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2016
Later among the works it cites.
Neurogenesis deep learning: Extending deep networks to accommodate new classes
Timothy J Draelos, Nadine E Miner, Christopher C Lamb, Jonathan A Cox, Craig M Vineyard, Kristofor D Carlson, William M Severa, Conrad D James, and James B Aimone · 2017
Closest in time.
Pathnet: Evolution channels gradient descent in super neural networks
Original
Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A Rusu, Alexander Pritzel, and Daan Wierstra · 2017
Closest in time.
Audio set: An ontology and human-labeled dataset for audio events
Jort F. Gemmeke, Daniel P. W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Channing Moore, Manoj Plakal, and Marvin Ritter · 2017
Closest in time.
Overcoming catastrophic forgetting in neural networks
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, et al · 2017
Closest in time.
Engrams and circuits crucial for systems consolidation of a memory
Takashi Kitamura, Sachie K Ogawa, Dheeraj S Roy, Teruhiro Okuyama, Mark D Morrissey, Lillian M Smith, Roger L Redondo, and Susumu Tonegawa · 2017
Closest in time.
Sleep is for forgetting
Gina R Poe · 2017
Closest in time.
iCaRL: Incremental classifier and representation learning
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, and Christoph H Lampert · 2017
Closest in time.
Measuring catastrophic forgetting in neural networks
Ronald Kemker, Marc McClure, Angelina Abitino, Tyler Hayes, and Christopher Kanan · 2018
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…