2022

Continual Learning with Foundation Models: An Empirical Study of Latent Replay

Ostapenko, Oleksiy, Lesort, Timothee, Rodríguez, Pau et al.

Understand

Rapid development of large-scale pre-training has resulted in foundation models that can act as effective feature extractors on a variety of downstream tasks and domains.

Motivated by this, we study the efficacy of pre-trained vision models as a foundation for downstream continual learning (CL) scenarios.
Our goal is twofold.
First, we want to understand the compute-accuracy trade-off between CL in the raw-data space and in the latent space of pre-trained encoders.

Built on

Continual learning in practice
Original
T. Diethe, T. Borchert, E. Thereska, B. d. B. Pigem, and N. Lawrence · 1903
Earlier work this paper cites.
Three scenarios for continual learning
Original
G. M. van de Ven and A. S. Tolias · 1904
Earlier work this paper cites.
Large scale incremental learning
Original
Y. Wu, Y. Chen, L. Wang, Y. Ye, Z. Liu, Y. Guo, and Y. Fu · 1905
Earlier work this paper cites.
Discorl: Continual reinforcement learning via policy distillation
Original
R. Traoré, H. Caselles-Dupré, T. Lesort, T. Sun, G. Cai, N. D. Rodríguez, and D. Filliat · 1907
Earlier work this paper cites.
Continual learning: A comparative study on how to defy forgetting in classification tasks, 2019
Original
M. D. Lange, R. Aljundi, M. Masana, S. Parisot, X. Jia, A. Leonardis, G. Slabaugh, and T. Tuytelaars · 1909
Earlier work this paper cites.
Online fast adaptation and knowledge accumulation: a new approach to continual learning
Original
M. Caccia, P. Rodriguez, O. Ostapenko, F. Normandin, M. Lin, L. Caccia, I. Laradji, I. Rish, A. Lacoste, D. Vazquez, and L. Charlin · 2003
Earlier work this paper cites.
Incremental linear discriminant analysis for classification of data streams
Shaoning Pang, S. Ozawa, and N. Kasabov · 2005
Earlier work this paper cites.
Insights from the Future for Continual Learning
Original
A. Douillard, E. Valle, C. Ollion, T. Robert, and M. Cord · 2006
Earlier work this paper cites.
Continual learning: Tackling catastrophic forgetting in deep neural networks with replay processes, 2020
Original
T. Lesort · 2007
Earlier work this paper cites.
Imagenet: A large-scale hierarchical image database
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei · 2009
Earlier work this paper cites.
Learning multiple layers of features from tiny images
A. Krizhevsky, G. Hinton, et al · 2009
Earlier work this paper cites.
Caltech-UCSD Birds 200
P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona · 2010
Earlier work this paper cites.
Cats and dogs
O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. Jawahar · 2012
Earlier work this paper cites.
3d object representations for fine-grained categorization
J. Krause, M. Stark, J. Deng, and L. Fei-Fei · 2013
Earlier work this paper cites.
Fine-grained visual classification of aircraft
S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi · 2013
Earlier work this paper cites.
Describing textures in the wild
M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, , and A. Vedaldi · 2014
Earlier work this paper cites.
How transferable are features in deep neural networks?
J. Yosinski, J. Clune, Y. Bengio, and H. Lipson · 2014
Earlier work this paper cites.
Batch normalization: Accelerating deep network training by reducing internal covariate shift
S. Ioffe and C. Szegedy · 2015
Earlier work this paper cites.
Faster r-cnn: Towards real-time object detection with region proposal networks
S. Ren, K. He, R. Girshick, and J. Sun · 2015
Earlier work this paper cites.
Wide residual networks
S. Zagoruyko and N. Komodakis · 2016
Earlier work this paper cites.
Memory aware synapses: Learning what (not) to forget
Original
R. Aljundi, F. Babiloni, M. Elhoseiny, M. Rohrbach, and T. Tuytelaars · 2017
Earlier work this paper cites.
Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs
L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille · 2017
Earlier work this paper cites.
Overcoming catastrophic forgetting in neural networks
J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, et al · 2017
Earlier work this paper cites.
Ubernet: Training a universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory
I. Kokkinos · 2017
Earlier work this paper cites.
Gradient episodic memory for continual learning
D. Lopez-Paz and M.-A. Ranzato · 2017
Earlier work this paper cites.
Revisiting unreasonable effectiveness of data in deep learning era
C. Sun, A. Shrivastava, S. Singh, and A. Gupta · 2017
Earlier work this paper cites.
Inception-v4, inception-resnet and the impact of residual connections on learning
C. Szegedy, S. Ioffe, V. Vanhoucke, and A. A. Alemi · 2017
Earlier work this paper cites.
Lifelong learning with dynamically expandable networks
Original
J. Yoon, E. Yang, J. Lee, and S. J. Hwang · 2017
Earlier work this paper cites.
Deesil: Deep-shallow incremental learning
E. Belouadah and A. Popescu · 2018
Earlier work this paper cites.
End-to-end incremental learning
Original
F. M. Castro, M. J. Marin-Jimenez, N. Guil, C. Schmid, and K. Alahari · 2018
Earlier work this paper cites.
Efficient lifelong learning with a-gem
Original
A. Chaudhry, M. Ranzato, M. Rohrbach, and M. Elhoseiny · 2018
Earlier work this paper cites.

Similar

Bert: Pre-training of deep bidirectional transformers for language understanding
Original
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova · 2018
Cited alongside, same era.
Memory efficient experience replay for streaming learning
Original
T. L. Hayes, N. D. Cahill, and C. Kanan · 2018
Cited alongside, same era.
Progress & compress: A scalable framework for continual learning
Original
J. Schwarz, J. Luketina, W. M. Czarnecki, A. Grabska-Barwinska, Y. W. Teh, R. Pascanu, and R. Hadsell · 2018
Cited alongside, same era.
Learning a unified classifier incrementally via rebalancing
S. Hou, X. Pan, C. C. Loy, Z. Wang, and D. Lin · 2019
Cited alongside, same era.
Meta-learning representations for continual learning
K. Javed and M. White · 2019
Cited alongside, same era.
Continual reinforcement learning deployed in real-life using policydistillation and sim2real transfer
R. T. Kalifou, H. Caselles-Dupré, T. Lesort, T. Sun, N. Diaz-Rodriguez, and D. Filliat · 2019
Cited alongside, same era.
Regularization shortcomings for continual learning
Original
T. Lesort, A. Stoian, and D. Filliat · 2019
Cited alongside, same era.
Toward understanding catastrophic forgetting in continual learning
Original
C. V. Nguyen, A. Achille, M. Lam, T. Hassner, V. Mahadevan, and S. Soatto · 2019
Cited alongside, same era.
On the information bottleneck theory of deep learning
A. M. Saxe, Y. Bansal, J. Dapello, M. Advani, A. Kolchinsky, B. D. Tracey, and D. D. Cox · 2019
Cited alongside, same era.
Efficientnet: Rethinking model scaling for convolutional neural networks
M. Tan and Q. Le · 2019
Cited alongside, same era.
Billion-scale semi-supervised learning for image classification
Original
I. Z. Yalniz, H. Jégou, K. Chen, M. Paluri, and D. Mahajan · 2019
Cited alongside, same era.
A comprehensive study of class incremental learning algorithms for visual tasks
E. Belouadah, A. Popescu, and I. Kanellos · 2020
Cited alongside, same era.

Then

Generative vs. discriminative: Rethinking the meta-continual learning
M. Banayeeanzade, R. Mirzaiezadeh, H. Hasani, and M. Soleymani · 2021
Later among the works it cites.
On the opportunities and risks of foundation models
Original
R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, et al · 2021
Later among the works it cites.
Special: Self-supervised pretraining for continual learning
L. Caccia and J. Pineau · 2021
Later among the works it cites.
Emerging properties in self-supervised vision transformers
Original
M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin · 2021
Later among the works it cites.
Revisiting rainbow: Promoting more insightful and inclusive deep reinforcement learning research
J. S. O. Ceron and P. S. Castro · 2021
Later among the works it cites.
Continual learning with differential privacy
P. Desai, P. Lai, N. Phan, and M. T. Thai · 2021
Later among the works it cites.
Continuum: Simple management of complex continual learning scenarios, 2021
Original
A. Douillard and T. Lesort · 2021
Later among the works it cites.
Dytox: Transformers for continual learning with dynamic token expansion
Original
A. Douillard, A. Ramé, G. Couairon, and M. Cord · 2021
Later among the works it cites.
On the role of neural collapse in transfer learning
Original
T. Galanti, A. György, and M. Hutter · 2021
Later among the works it cites.
Self-supervised training enhances online continual learning
J. Gallardo, T. L. Hayes, and C. Kanan · 2021
Later among the works it cites.
Towards continual knowledge learning of language models
Original
J. Jang, S. Ye, S. Yang, J. Shin, J. Han, G. Kim, S. J. Choi, and M. Seo · 2021
Later among the works it cites.
Continual learning in the teacher-student setup: Impact of task similarity
S. Lee, S. Goldt, and A. Saxe · 2021
Later among the works it cites.
Lifelong learning of compositional structures
J. A. Mendez and E. EATON · 2021
Later among the works it cites.
Continual learning via local module composition
O. Ostapenko, P. Rodriguez, M. Caccia, and L. Charlin · 2021
Later among the works it cites.
Learning transferable visual models from natural language supervision
Original
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al · 2021
Later among the works it cites.
Anatomy of catastrophic forgetting: Hidden representations and task semantics
V. V. Ramasesh, E. Dyer, and M. Raghu · 2021
Later among the works it cites.
Dall-e: Creating images from text
M. D. M. Reddy, M. S. M. Basha, M. M. C. Hari, and M. N. Penchalaiah · 2021
Later among the works it cites.
Imagenet-21k pretraining for the masses
Original
T. Ridnik, E. Ben-Baruch, A. Noy, and L. Zelnik-Manor · 2021
Later among the works it cites.
Multitask prompted training enables zero-shot task generalization
Original
V. Sanh, A. Webson, C. Raffel, S. H. Bach, L. Sutawika, Z. Alyafeai, A. Chaffin, A. Stiegler, T. L. Scao, A. Raja, et al · 2021
Later among the works it cites.
Training data-efficient image transformers & distillation through attention
H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou · 2021
Later among the works it cites.
Efficient continual learning with modular networks and task-driven priors
T. Veniat, L. Denoyer, and M. Ranzato · 2021
Later among the works it cites.
Privacy-preserving machine learning: Methods, challenges and directions
Original
R. Xu, N. Baracaldo, and J. Joshi · 2021
Later among the works it cites.
Florence: A new foundation model for computer vision
Original
L. Yuan, D. Chen, Y.-L. Chen, N. Codella, X. Dai, J. Gao, H. Hu, X. Huang, B. Li, C. Li, et al · 2021
Later among the works it cites.
Head2toe: Utilizing intermediate representations for better transfer learning
Original
U. Evci, V. Dumoulin, H. Larochelle, and M. C. Mozer · 2022
Closest in time.
How well does self-supervised pre-training perform with streaming data?
D. Hu, S. Yan, Q. Lu, L. HONG, H. Hu, Y. Zhang, Z. Li, X. Wang, and J. Feng · 2022
Closest in time.
An empirical investigation of the role of pre-training in lifelong learning, 2022
S. V. Mehta, D. Patil, S. Chandar, and E. Strubell · 2022
Closest in time.
Effect of scale on catastrophic forgetting in neural networks
V. V. Ramasesh, A. Lewkowycz, and E. Dyer · 2022
Closest in time.
Pushing the limits of self-supervised resnets: Can we outperform supervised learning without labels on imagenet?
Original
N. Tomasev, I. Bica, B. McWilliams, L. Buesing, R. Pascanu, C. Blundell, and J. Mitrovic · 2022
Closest in time.
What can we learn from misclassified imagenet images?
Original
S. Wen, A. S. Rios, K. Lekkala, and L. Itti · 2022
Closest in time.
Pretrained language model in continual learning: A comparative study
T. Wu, M. Caccia, Z. Li, Y.-F. Li, G. Qi, and G. Haffari · 2022
Closest in time.
Lit: Zero-shot transfer with locked-image text tuning
Original
X. Zhai, X. Wang, B. Mustafa, A. Steiner, D. Keysers, A. Kolesnikov, and L. Beyer · 2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…