Towards Understanding Multi-Task Learning (Generalization) of LLMs via Detecting and Exploring Task-Specific Neurons

Recursive deep models for semantic compositionality over a sentiment treebank

Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. 2013 · 2013

Earlier work this paper cites.

Deep inside convolutional networks: Visualising image classification models and saliency maps

Original

Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2014 · 2014

Earlier work this paper cites.

Teaching machines to read and comprehend

Karl Moritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015 · 2015

Earlier work this paper cites.

Learning to generate reviews and discovering sentiment

Original

Alec Radford, Rafal Józefowicz, and Ilya Sutskever. 2017 · 2017

Earlier work this paper cites.

Crowdsourcing multiple choice science questions

Johannes Welbl, Nelson F. Liu, and Matt Gardner. 2017 · 2017

Earlier work this paper cites.

Memory aware synapses: Learning what (not) to forget

Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. 2018 · 2018

Earlier work this paper cites.

Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization

Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018 · 2018

Earlier work this paper cites.

A broad-coverage challenge corpus for sentence understanding through inference

Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2018 · 2018

Earlier work this paper cites.

GLUE: A multi-task benchmark and analysis platform for natural language understanding

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019 · 2019

Earlier work this paper cites.

TWEETQA: A social media focused question answering dataset

Wenhan Xiong, Jiawei Wu, Hong Wang, Vivek Kulkarni, Mo Yu, Shiyu Chang, Xiaoxiao Guo, and William Yang Wang. 2019 · 2019

Earlier work this paper cites.

PAWS: paraphrase adversaries from word scrambling

Yuan Zhang, Jason Baldridge, and Luheng He. 2019 · 2019

Earlier work this paper cites.

Language models are few-shot learners

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020 · 2020

Earlier work this paper cites.

Evaluating the state-of-the-art of end-to-end natural language generation: The E2E NLG challenge

Ondrej Dusek, Jekaterina Novikova, and Verena Rieser. 2020 · 2020

Earlier work this paper cites.

The multilingual amazon reviews corpus

Phillip Keung, Yichao Lu, György Szarvas, and Noah A. Smith. 2020 · 2020

Earlier work this paper cites.

Commongen: A constrained text generation challenge for generative commonsense reasoning

Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. 2020 · 2020

Earlier work this paper cites.

Generative memory for lifelong learning

Xin Su, Shangqi Guo, Tian Tan, and Feng Chen. 2020 · 2020

Earlier work this paper cites.

Continual learning with deep architectures

Irina Rish. 2021 · 2021

Earlier work this paper cites.

Importance-based neuron allocation for multilingual neural machine translation

Wanying Xie, Yang Feng, Shuhao Gu, and Dong Yu. 2021 · 2021

Cited alongside, same era.

Knowledge neurons in pretrained transformers

Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. 2022 · 2022

Cited alongside, same era.

State-of-the-art generalisation research in NLP: a taxonomy and review

Original

Dieuwke Hupkes, Mario Giulianelli, Verna Dankers, Mikel Artetxe, Yanai Elazar, Tiago Pimentel, Christos Christodoulopoulos, Karim Lasri, Naomi Saphra, Arabella Sinclair, Dennis Ulmer, Florian Schottmann, Khuyagbaatar Batsuren, Kaiser Sun, Koustuv Sinha, Leila Khalatbari, Maria Ryskina, Rita Frieske, Ryan Cotterell, and Zhijing Jin. 2022 · 2022

Cited alongside, same era.

Continual learning of natural language processing tasks: A survey

Original

Zixuan Ke and Bing Liu. 2022 · 2022

Cited alongside, same era.

Multitask prompted training enables zero-shot task generalization

Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Févry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. 2022 · 2022

Orthogonal subspace learning for language model continual learning

Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, and Xuanjing Huang. 2023a · 2023

Later among the works it cites.

Rehearsal-free continual language learning via efficient parameter isolation

Zhicheng Wang, Yufang Liu, Tao Ji, Xiaoling Wang, Yuanbin Wu, Congcong Jiang, Ye Chao, Zhencong Han, Ling Wang, Xu Shao, and Wenqiu Zeng. 2023b · 2023

Later among the works it cites.

DEPN: detecting and editing privacy neurons in pretrained language models

Xinwei Wu, Junzhuo Li, Minghui Xu, Weilong Dong, Shuangzhi Wu, Chao Bian, and Deyi Xiong. 2023a · 2023

Later among the works it cites.

LIMA: less is more for alignment

Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. 2023 · 2023

Later among the works it cites.

Language models can exploit cross-task in-context learning for data-scarce novel tasks

Anwoy Chatterjee, Eshaan Tanwar, Subhabrata Dutta, and Tanmoy Chakraborty. 2024 · 2024

Closest in time.