Fetching the paper…
Reading the bibliography…
Continual learning (CL) in large language models (LLMs) is an evolving domain that focuses on developing efficient and sustainable training strategies to adapt models to emerging knowledge and achieve robustness in dynamic environments.
Sentence-bert: Sentence embeddings using siamese bert-networks
Nils Reimers and Iryna Gurevych · 1908
Earlier work this paper cites.
Catastrophic forgetting in connectionist networks
Robert M French · 1999
Earlier work this paper cites.
Learning multiple layers of features from tiny images
Alex Krizhevsky, Geoffrey Hinton, et al · 2009
Earlier work this paper cites.
Adam: A method for stochastic optimization
Diederik Kingma and Jimmy Ba · 2015
Earlier work this paper cites.
Tiny imagenet visual recognition challenge
Yann Le and Xuan Yang · 2015
Earlier work this paper cites.
Squad: 100,000+ questions for machine comprehension of text
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang · 2016
Earlier work this paper cites.
Bert: Pre-training of deep bidirectional transformers for language understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova · 2018
Earlier work this paper cites.
Openwebtext corpus
Aaron Gokaslan and Vanya Cohen · 2019
Earlier work this paper cites.
S2orc: The semantic scholar open research corpus
Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Dan S Weld · 2019
Earlier work this paper cites.
Language models are unsupervised multitask learners
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al · 2019
Earlier work this paper cites.
Sentence-bert: Sentence embeddings using siamese bert-networks
N Reimers · 2019
Earlier work this paper cites.
Lamol: Language modeling for lifelong language learning
Fan-Keng Sun, Cheng-Hao Ho, and Hung-Yi Lee · 2019
Earlier work this paper cites.
Bert post-training for review reading comprehension and aspect-based sentiment analysis
Hu Xu, Bing Liu, Lei Shu, and Philip S Yu · 2019
Earlier work this paper cites.
Continual lifelong learning in natural language processing: A survey
Magdalena Biesialska, Katarzyna Biesialska, and Marta R Costa-Jussa · 2020
Earlier work this paper cites.
Language models are few-shot learners
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al · 2020
Earlier work this paper cites.
Don’t stop pretraining: Adapt language models to domains and tasks
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith · 2020
Earlier work this paper cites.
Scaling laws for neural language models
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei · 2020
Cited alongside, same era.
Roberta: A robustly optimized bert pretraining approach, 2020
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov · 2020
Cited alongside, same era.
Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He · 2020
Cited alongside, same era.
Explaining neural scaling laws
Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma · 2021
Cited alongside, same era.
Demix layers: Disentangling domains for modular language modeling
Lifelong language pretraining with distribution-specialized experts
Wuyang Chen, Yanqi Zhou, Nan Du, Yanping Huang, James Laudon, Zhifeng Chen, and Claire Cui · 2023
Later among the works it cites.
Towards robust and efficient continual language learning
Adam Fisch, Amal Rannen-Triki, Razvan Pascanu, Jörg Bornschein, Angeliki Lazaridou, Elena Gribovskaya, and Marc’Aurelio Ranzato · 2023
Later among the works it cites.
Dissecting recall of factual associations in auto-regressive language models
Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson · 2023
Later among the works it cites.
Continual pre-training of large language models: How to (re) warm your model?
Kshitij Gupta, Benjamin Thérien, Adam Ibrahim, Mats L Richter, Quentin Anthony, Eugene Belilovsky, Irina Rish, and Timothée Lesort · 2023
Later among the works it cites.
An empirical investigation of the role of pre-training in lifelong learning
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Suchin Gururangan, Mike Lewis, Ari Holtzman, Noah A Smith, and Luke Zettlemoyer · 2021
Cited alongside, same era.
Lifelong pretraining: Continually adapting language models to emerging corpora
Xisen Jin, Dejiao Zhang, Henghui Zhu, Wei Xiao, Shang-Wen Li, Xiaokai Wei, Andrew Arnold, and Xiang Ren · 2021
Cited alongside, same era.
Effect of scale on catastrophic forgetting in neural networks
Vinay Venkatesh Ramasesh, Aitor Lewkowycz, and Ethan Dyer · 2021
Cited alongside, same era.
Pretrained language model in continual learning: A comparative study
Tongtong Wu, Massimo Caccia, Zhuang Li, Yuan-Fang Li, Guilin Qi, and Gholamreza Haffari · 2021
Cited alongside, same era.
Lifelong language learning with adapter based transformers
Shadab Khan, Surbhi Agarwal, and PK Srijith · 2022
Cited alongside, same era.
Estimating the carbon footprint of bloom, a 176b parameter language model, 2022
Alexandra Sasha Luccioni, Sylvain Viguier, and Anne-Laure Ligozat · 2022
Cited alongside, same era.
Elle: Efficient lifelong pre-training for emerging data
Yujia Qin, Jiajie Zhang, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou · 2022
Cited alongside, same era.
M2d2: A massively multi-domain language modeling dataset
Machel Reid, Victor Zhong, Suchin Gururangan, and Luke Zettlemoyer · 2022
Cited alongside, same era.
Sanket Vaibhav Mehta, Darshan Patil, Sarath Chandar, and Emma Strubell · 2023
Later among the works it cites.
Progressive prompts: Continual learning for language models
Anastasia Razdaibiedina, Yuning Mao, Rui Hou, Madian Khabsa, Mike Lewis, and Amjad Almahairi · 2023
Later among the works it cites.
Orthogonal subspace learning for language model continual learning
Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, and Xuanjing Huang · 2023
Later among the works it cites.
Continual memorization of factoids in large language models
Howard Chen, Jiayi Geng, Adithya Bhaskar, Dan Friedman, and Danqi Chen · 2024
Closest in time.
Continual pre-training mitigates forgetting in language and vision
Andrea Cossu, Antonio Carta, Lucia Passaro, Vincenzo Lomonaco, Tinne Tuytelaars, and Davide Bacciu · 2024
Closest in time.
Domain-adaptative continual learning for low-resource tasks: Evaluation on nepali
Sharad Duwal, Suraj Prasai, and Suresh Manandhar · 2024
Closest in time.
Gradient localization improves lifelong pretraining of language models
Jared Fernandez, Yonatan Bisk, and Emma Strubell · 2024
Closest in time.
Continual learning under language shift
Evangelia Gogoulou, Timothée Lesort, Magnus Boman, and Joakim Nivre · 2024
Closest in time.
Adapting large language models to log analysis with interpretable domain knowledge
Yuhe Ji, Yilun Liu, Feiyu Yao, Minggui He, Shimin Tao, Xiaofeng Zhao, Su Chang, Xinhua Yang, Weibin Meng, Yuming Xie, et al · 2024
Closest in time.
Redwhale: An adapted korean llm through efficient continual pretraining
Anh-Dung Vo, Minseong Jung, Wonbeen Lee, and Daewoo Choi · 2024
Closest in time.
Continual learning for large language models: A survey
Tongtong Wu, Linhao Luo, Yuan-Fang Li, Shirui Pan, Thuy-Trang Vu, and Gholamreza Haffari · 2024
Closest in time.