Fetching the paper…
Reading the bibliography…
Fine-tuning large models is highly effective, however, inference can be expensive and produces carbon emissions.
Language models are few-shot learners
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020 · 1901
Earlier work this paper cites.
Roberta: A robustly optimized bert pretraining approach
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019 · 1907
Earlier work this paper cites.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019 · 1910
Earlier work this paper cites.
A theory of the allocation of time
Gary S. Becker. 1965 · 1965
Earlier work this paper cites.
A new approach to consumer theory
Kelvin J. Lancaster. 1966 · 1966
Earlier work this paper cites.
Distilling the knowledge in a neural network
Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. 2015 · 2015
Earlier work this paper cites.
MS MARCO: A human generated machine reading comprehension dataset
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016 · 2016
Earlier work this paper cites.
FEVER: a large-scale dataset for fact extraction and VERification
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018 · 2018
Earlier work this paper cites.
Bert: Pre-training of deep bidirectional transformers for language understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 · 2019
Earlier work this paper cites.
Natural questions: A benchmark for question answering research
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019 · 2019
Earlier work this paper cites.
Are sixteen heads really better than one?
Paul Michel, Omer Levy, and Graham Neubig. 2019 · 2019
Earlier work this paper cites.
Partial or complete, that’s the question
Qiang Ning, Hangfeng He, Chuchu Fan, and Dan Roth. 2019 · 2019
Earlier work this paper cites.
Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019 · 2019
Earlier work this paper cites.
Huggingface’s transformers: State-of-the-art natural language processing
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2019 · 2019
Earlier work this paper cites.
Be your own teacher: Improve the performance of convolutional neural networks via self distillation
Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. 2019 · 2019
Earlier work this paper cites.
Retrieval augmented language model pre-training
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020 · 2020
Earlier work this paper cites.
Handling anomalies of synthetic questions in unsupervised question answering
Giwon Hong, Junmo Kang, Doyeon Lim, and Sung-Hyon Myaeng. 2020 · 2020
Earlier work this paper cites.
Characterising bias in compressed models
Sara Hooker, Nyalleng Moorosi, Gregory Clark, Samy Bengio, and Emily Denton. 2020 · 2020
Earlier work this paper cites.
Tinybert: Distilling bert for natural language understanding
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020 · 2020
Earlier work this paper cites.
Regularization of distinct strategies for unsupervised question generation
Junmo Kang, Giwon Hong, Haritz Puerto San Roman, and Sung-Hyon Myaeng. 2020 · 2020
Earlier work this paper cites.
Language models as fact checkers?
Nayeon Lee, Belinda Z. Li, Sinong Wang, Wen-tau Yih, Hao Ma, and Madian Khabsa. 2020 · 2020
Earlier work this paper cites.
Exploring the limits of transfer learning with a unified text-to-text transformer
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020 · 2020
Cited alongside, same era.
How much knowledge can you pack into the parameters of a language model?
Adam Roberts, Colin Raffel, and Noam Shazeer. 2020 · 2020
Cited alongside, same era.
WNUT-2020 task 1 overview: Extracting entities and relations from wet lab protocols
Jeniya Tabassum, Wei Xu, and Alan Ritter. 2020 · 2020
Cited alongside, same era.
Bertscore: Evaluating text generation with bert
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020 · 2020
Cited alongside, same era.
Bert learns to teach: Knowledge distillation with meta learning
Wangchunshu Zhou, Canwen Xu, and Julian McAuley. 2022b · 2020
Cited alongside, same era.
Pareto-optimal quantized resnet is mostly 4-bit
Calibrate before use: Improving few-shot performance of language models
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021 · 2021
Later among the works it cites.
Why knowledge distillation amplifies gender bias and how to mitigate from the perspective of DistilBERT
Jaimeen Ahn, Hwaran Lee, Jinhwa Kim, and Alice Oh. 2022 · 2022
Later among the works it cites.
Clean or annotate: How to spend a limited data collection budget
Derek Chen, Zhou Yu, and Samuel Bowman. 2022 · 2022
Later among the works it cites.
Improving large-scale paraphrase acquisition and generation
Yao Dou, Chao Jiang, and Wei Xu. 2022 · 2022
Later among the works it cites.
Knowledge distillation for sustainable neural machine translation
Wandri Jooste, Andy Way, Rejwanul Haque, and Riccardo Superbo. 2022 · 2022
Later among the works it cites.
A few more examples may be worth billions of parameters
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
AmirAli Abdolrashidi, Lisa Wang, Shivani Agrawal, Jonathan Malmaud, Oleg Rybakov, Chas Leichner, and Lukasz Lew. 2021 · 2021
Cited alongside, same era.
Pre-train or annotate? domain adaptation with a constrained budget
Fan Bai, Alan Ritter, and Wei Xu. 2021 · 2021
Cited alongside, same era.
Knowledge distillation: A survey
Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. 2021 · 2021
Cited alongside, same era.
Cuad: An expert-annotated nlp dataset for legal contract review
Dan Hendrycks, Collin Burns, Anya Chen, and Spencer Ball. 2021 · 2021
Cited alongside, same era.
Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. 2021 · 2021
Cited alongside, same era.
How to train BERT with an academic budget
Peter Izsak, Moshe Berchansky, and Omer Levy. 2021 · 2021
Cited alongside, same era.
The power of scale for parameter-efficient prompt tuning
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021 · 2021
Cited alongside, same era.
Yuval Kirstain, Patrick Lewis, Sebastian Riedel, and Omer Levy. 2022 · 2022
Later among the works it cites.
Few-shot anaphora resolution in scientific protocols via mixtures of in-context experts
Nghia Le, Fan Bai, and Alan Ritter. 2022 · 2022
Later among the works it cites.
Knowledge distillation with reptile meta-learning for pretrained language model compression
Xinge Ma, Jin Wang, Liang-Chih Yu, and Xuejie Zhang. 2022 · 2022
Later among the works it cites.
Teaching small language models to reason
Lucie Charlotte Magister, Jonathan Mallinson, Jakub Adamek, Eric Malmi, and Aliaksei Severyn. 2022 · 2022
Later among the works it cites.
Noisy channel language model prompting for few-shot text classification
Sewon Min, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022 · 2022
Later among the works it cites.
WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models
Benjamin Minixhofer, Fabian Paischer, and Navid Rekabsaz. 2022 · 2022
Later among the works it cites.
Training language models to follow instructions with human feedback
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022 · 2022
Later among the works it cites.
Predicting attention sparsity in transformers
Marcos Treviso, António Góis, Patrick Fernandes, Erick Fonseca, and André FT Martins. 2022 · 2022
Later among the works it cites.
Chain of thought prompting elicits reasoning in large language models
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. 2022 · 2022
Later among the works it cites.
Structured pruning learns compact and accurate models
Mengzhou Xia, Zexuan Zhong, and Danqi Chen. 2022 · 2022
Later among the works it cites.
A survey on model compression and acceleration for pretrained language models
Canwen Xu and Julian McAuley. 2022 · 2022
Later among the works it cites.
Sparse distillation: Speeding up text classification by using bigger student models
Qinyuan Ye, Madian Khabsa, Mike Lewis, Sinong Wang, Xiang Ren, and Aaron Jaech. 2022 · 2022
Later among the works it cites.
Stanceosaurus: Classifying stance towards multilingual misinformation
Jonathan Zheng, Ashutosh Baheti, Tarek Naous, Wei Xu, and Alan Ritter. 2022 · 2022
Later among the works it cites.
Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. 2023 · 2023
Closest in time.
Principle-driven self-alignment of language models from scratch with minimal human supervision
Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. 2023 · 2023
Closest in time.