Unlocking Continual Learning Abilities in Language Models

Language models are few-shot learners

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020 · 1901

Earlier work this paper cites.

Roberta: A robustly optimized bert pretraining approach

Original

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019 · 1907

Earlier work this paper cites.

Winogrande: An adversarial winograd schema challenge at scale

Original

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019 · 1907

Earlier work this paper cites.

Exploring the limits of transfer learning with a unified text-to-text transformer

Original

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2023 · 1910

Earlier work this paper cites.

Catastrophic interference in connectionist networks: The sequential learning problem

Michael McCloskey and Neal J Cohen. 1989 · 1989

Earlier work this paper cites.

Learning capability and storage capacity of two-hidden-layer feedforward networks

Guang-Bin Huang. 2003 · 2003

Earlier work this paper cites.

Dark experience for general continual learning: a strong, simple baseline

Original

Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. 2020 · 2004

Earlier work this paper cites.

Mining and summarizing customer reviews

Minqing Hu and Bing Liu. 2004 · 2004

Earlier work this paper cites.

A holistic lexicon-based approach to opinion mining

Xiaowen Ding, Bing Liu, and Philip S Yu. 2008 · 2008

Earlier work this paper cites.

Learning word vectors for sentiment analysis

Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011 · 2011

Earlier work this paper cites.

Character-level convolutional networks for text classification

Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015 · 2015

Earlier work this paper cites.

Chemprot-3.0: a global chemical biology diseases mapping

Jens Kringelum, Sonny Kim Kjaerulff, Søren Brunak, Ole Lund, Tudor I Oprea, and Olivier Taboureau. 2016 · 2016

Earlier work this paper cites.

Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence , page 556–572

Arslan Chaudhry, Puneet K. Dokania, Thalaiyasingam Ajanthan, and Philip H. S. Torr. 2018 · 2018

Earlier work this paper cites.

Think you have solved question answering? try arc, the ai2 reasoning challenge

Original

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018 · 2018

Earlier work this paper cites.

Measuring the evolution of a scientific field through citation frames

David Jurgens, Srijan Kumar, Raine Hoover, Daniel A. McFarland, and Dan Jurafsky. 2018 · 2018

Earlier work this paper cites.

Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction

Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. 2018 · 2018

Earlier work this paper cites.

Overcoming catastrophic forgetting with hard attention to the task

Joan Serrà, Didac Suris, Marius Miron, and Alexandros Karatzoglou. 2018 · 2018

Earlier work this paper cites.

Glue: A multi-task benchmark and analysis platform for natural language understanding

Original

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018 · 2018

Earlier work this paper cites.

Episodic memory in lifelong language learning

Cyprien de Masson D’Autume, Sebastian Ruder, Lingpeng Kong, and Dani Yogatama. 2019 · 2019

Earlier work this paper cites.

The lottery ticket hypothesis: Finding sparse, trainable neural networks

Original

Jonathan Frankle and Michael Carbin. 2019 · 2019

Cited alongside, same era.

Parameter-efficient transfer learning for nlp

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019 · 2019

Cited alongside, same era.

Justifying recommendations using distantly-labeled reviews and fine-grained aspects

Jianmo Ni, Jiacheng Li, and Julian J. McAuley. 2019 · 2019

Cited alongside, same era.

BERT post-training for review reading comprehension and aspect-based sentiment analysis

Hu Xu, Bing Liu, Lei Shu, and Philip S. Yu. 2019 · 2019

Cited alongside, same era.

Hellaswag: Can a machine really finish your sentence?

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019 · 2019

Cited alongside, same era.

Wizardcoder: Empowering code large language models with evol-instruct

Original

Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023 · 2023

Later among the works it cites.

Emergent mixture-of-experts: Can dense pre-trained transformers benefit from emergent modular structures?

Original

Zihan Qiu, Zeyu Huang, and Jie Fu. 2023 · 2023

Later among the works it cites.

Progressive prompts: Continual learning for language models

Anastasia Razdaibiedina, Yuning Mao, Rui Hou, Madian Khabsa, Mike Lewis, and Amjad Almahairi. 2023 · 2023

Later among the works it cites.

Llama: Open and efficient foundation language models

Original

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023 · 2023

Later among the works it cites.

Orthogonal subspace learning for language model continual learning