Fetching the paper…
Reading the bibliography…
Pre-trained models are nowadays a fundamental component of machine learning research.
Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem
McCloskey Michael, Cohen Neal J · 1989
Earlier work this paper cites.
Catastrophic Forgetting in Connectionist Networks
French Robert · 1999
Earlier work this paper cites.
Deep Residual Learning for Image Recognition
He Kaiming, Zhang Xiangyu, Ren Shaoqing, Sun Jian · 2016
Earlier work this paper cites.
Pointer Sentinel Mixture Models
Merity Stephen, Xiong Caiming, Bradbury James, Socher Richard · 2016
Earlier work this paper cites.
CORe50: A New Dataset and Benchmark for Continuous Object Recognition
Lomonaco Vincenzo, Maltoni Davide · 2017
Earlier work this paper cites.
Gradient Episodic Memory for Continual Learning
Lopez-Paz David, Ranzato Marc’Aurelio · 2017
Earlier work this paper cites.
Attention Is All You Need
Vaswani Ashish, Shazeer Noam, Parmar Niki, Uszkoreit Jakob, Jones Llion, Gomez Aidan N, Kaiser Łukasz, Polosukhin Illia · 2017
Earlier work this paper cites.
SentEval: An Evaluation Toolkit for Universal Sentence Representations
Conneau Alexis, Kiela Douwe · 2018
Earlier work this paper cites.
The INaturalist Species Classification and Detection Dataset
Van Horn Grant, Mac Aodha Oisin, Song Yang, Cui Yin, Sun Chen, Shepard Alex, Adam Hartwig, Perona Pietro, Belongie Serge · 2018
Earlier work this paper cites.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin Jacob, Chang Ming-Wei, Lee Kenton, Toutanova Kristina · 2019
Earlier work this paper cites.
ArXiV Archive: A Tidy and Complete Archive of Metadata for Papers on Arxiv.Org, 1993-2019. 2019
Geiger R. Stuart · 2019
Earlier work this paper cites.
Similarity of Neural Network Representations Revisited
Kornblith Simon, Norouzi Mohammad, Lee Honglak, Hinton Geoffrey · 2019
Earlier work this paper cites.
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu Yinhan, Ott Myle, Goyal Naman, Du Jingfei, Joshi Mandar, Chen Danqi, Levy Omer, Lewis Mike, Zettlemoyer Luke, Stoyanov Veselin · 2019
Earlier work this paper cites.
Continual Lifelong Learning with Neural Networks: A Review
Parisi German I, Kemker Ronald, Part Jose L, Kanan Christopher, Wermter Stefan · 2019
Earlier work this paper cites.
Transfer Learning in Natural Language Processing
Ruder Sebastian, Peters Matthew E., Swayamdipta Swabha, Wolf Thomas · 2019
Earlier work this paper cites.
Improved Baselines with Momentum Contrastive Learning
Chen Xinlei, Fan Haoqi, Girshick Ross, He Kaiming · 2020
Cited alongside, same era.
An Image Is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy Alexey, Beyer Lucas, Kolesnikov Alexander, Weissenborn Dirk, Zhai Xiaohua, Unterthiner Thomas, Dehghani Mostafa, Minderer Matthias, Heigold Georg, Gelly Sylvain, Uszkoreit Jakob, Houlsby Neil · 2020
Cited alongside, same era.
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Gururangan Suchin, Marasović Ana, Swayamdipta Swabha, Lo Kyle, Beltagy Iz, Downey Doug, Smith Noah A · 2020
Cited alongside, same era.
Embracing Change: Continual Learning in Deep Neural Networks
Hadsell Raia, Rao Dushyant, Rusu Andrei A, Pascanu Razvan · 2020
Cited alongside, same era.
BioBERT: A Pre-Trained Biomedical Language Representation Model for Biomedical Text Mining
Lee Jinhyuk, Yoon Wonjin, Kim Sungdong, Kim Donghyeon, Kim Sunkyu, So Chan Ho, Kang Jaewoo · 2020
Cited alongside, same era.
Towards Continual Knowledge Learning of Language Models
Jang Joel, Ye Seonghyeon, Yang Sohee, Shin Joongbo, Han Janghoon, Kim Gyeonghun, Choi Stanley Jungkyu, Seo Minjoon · 2021
Later among the works it cites.
Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora
Jin Xisen, Zhang Dejiao, Zhu Henghui, Xiao Wei, Li Shang-Wen, Wei Xiaokai, Arnold Andrew, Ren Xiang · 2021
Later among the works it cites.
Mind the Gap: Assessing Temporal Generalization in Neural Language Models
Lazaridou Angeliki, Kuncoro Adhiguna, Gribovskaya Elena, Agrawal Devang, Liska Adam, Terzi Tayfun, Gimenez Mai, d’Autume Cyprien de Masson, Kočiský Tomáš, Ruder Sebastian, Yogatama Dani, Cao Kris, Young Susannah, Blunsom Phil · 2021
Later among the works it cites.
Avalanche: An End-to-End Library for Continual Learning
Lomonaco Vincenzo, Pellegrini Lorenzo, Cossu Andrea, Carta Antonio, Graffieti Gabriele, Hayes Tyler L., De Lange Matthias, Masana Marc, Pomponi Jary, van de Ven Gido, Mundt Martin, She Qi, Cooper Keiland, Forest Jeremy, Belouadah Eden, Calderara Simone, Parisi German I., Cuzzolin Fabio, Tolias Andreas, Scardapane Simone, Antiga Luca, Amhad Subutai, Popescu Adrian, Kanan Christopher, van de Weijer Joost, Tuytelaars Tinne, Bacciu Davide, Maltoni Davide · 2021
Later among the works it cites.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Continual Learning for Robotics: Definition, Framework, Learning Strategies, Opportunities and Challenges
Lesort Timothée, Lomonaco Vincenzo, Stoian Andrei, Maltoni Davide, Filliat David, Díaz-Rodríguez Natalia · 2020
Cited alongside, same era.
Bayesian Nonparametric Weight Factorization for Continual Learning
Mehta Nikhil, Liang Kevin J, Carin Lawrence · 2020
Cited alongside, same era.
Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth
Nguyen Thao, Raghu Maithra, Kornblith Simon · 2020
Cited alongside, same era.
Multi-Stage Pre-training for Low-Resource Domain Adaptation
Zhang Rong, Gangi Reddy Revanth, Sultan Md Arafat, Castelli Vittorio, Ferritto Anthony, Florian Radu, Sarioglu Kayi Efsun, Roukos Salim, Sil Avi, Ward Todd · 2020
Cited alongside, same era.
BEiT: BERT Pre-Training of Image Transformers
Bao Hangbo, Dong Li, Piao Songhao, Wei Furu · 2021
Cited alongside, same era.
A Continual Learning Survey: Defying Forgetting in Classification Tasks
De Lange Matthias, Aljundi Rahaf, Masana Marc, Parisot Sarah, Jia Xu, Leonardis Ales, Slabaugh Gregory, Tuytelaars Tinne · 2021
Cited alongside, same era.
Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing
Gu Yu, Tinn Robert, Cheng Hao, Lucas Michael, Usuyama Naoto, Liu Xiaodong, Naumann Tristan, Gao Jianfeng, Poon Hoifung · 2021
Cited alongside, same era.
Representational Continuity for Unsupervised Continual Learning
Madaan Divyam, Yoon Jaehong, Li Yuanchun, Liu Yunxin, Hwang Sung Ju · 2021
Later among the works it cites.
An Empirical Investigation of the Role of Pre-training in Lifelong Learning
Mehta Sanket Vaibhav, Patil Darshan, Chandar Sarath, Strubell Emma · 2021
Later among the works it cites.
Effect of Scale on Catastrophic Forgetting in Neural Networks
Ramasesh Vinay Venkatesh, Lewkowycz Aitor, Dyer Ethan · 2021
Later among the works it cites.
Continual Domain-Tuning for Pretrained Language Models
Rongali Subendhu, Jagannatha Abhyuday, Rawat Bhanu Pratap Singh, Yu Hong · 2021
Later among the works it cites.
Pretrained Language Model in Continual Learning: A Comparative Study
Wu Tongtong, Caccia Massimo, Li Zhuang, Li Yuan-Fang, Qi Guilin, Haffari Gholamreza · 2021
Later among the works it cites.
Probing Representation Forgetting in Supervised and Unsupervised Continual Learning
Davari MohammadReza, Asadi Nader, Mudur Sudhir, Aljundi Rahaf, Belilovsky Eugene · 2022
Closest in time.
DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion
Douillard Arthur, Ramé Alexandre, Couairon Guillaume, Cord Matthieu · 2022
Closest in time.
TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Models
Jang Joel, Ye Seonghyeon, Lee Changho, Yang Sohee, Shin Joongbo, Han Janghoon, Kim Gyeonghun, Seo Minjoon · 2022
Closest in time.
TimeLMs: Diachronic Language Models from Twitter
Loureiro Daniel, Barbieri Francesco, Neves Leonardo, Anke Luis Espinosa, Camacho-Collados Jose · 2022
Closest in time.
ELLE: Efficient Lifelong Pre-training for Emerging Data
Qin Yujia, Zhang Jiajie, Lin Yankai, Liu Zhiyuan, Li Peng, Sun Maosong, Zhou Jie · 2022
Closest in time.