A Survey of Small Language Models

Tinybert: Distilling bert for natural language understanding

Original

Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2019 · 1909

Earlier work this paper cites.

Distilbert, a distilled version of bert: Smaller, faster, cheaper and lighter

Original

V Sanh. 2019 · 1910

Earlier work this paper cites.

Über Die Praktische Auflösung von Integralgleichungen mit Anwendungen auf Randwertaufgaben

E. J. Nyström. 1930 · 1930

Earlier work this paper cites.

Nas-bert: task-agnostic and adaptive-size bert compression with neural architecture search

Jin Xu, Xu Tan, Renqian Luo, Kaitao Song, Jian Li, Tao Qin, and Tie-Yan Liu. 2021 · 1943

Earlier work this paper cites.

CrowS-pairs: A challenge dataset for measuring social biases in masked language models

Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. 2020 · 1967

Earlier work this paper cites.

Reformer: The efficient transformer

Original

Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. 2020 · 2001

Earlier work this paper cites.

Longformer: The long-document transformer

Original

Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020 · 2004

Earlier work this paper cites.

Linformer: Self-attention with linear complexity

Original

Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. 2020a · 2006

Earlier work this paper cites.

Adam: A method for stochastic optimization

P Kingma Diederik. 2014 · 2014

Earlier work this paper cites.

Distilling the knowledge in a neural network

Original

Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. 2015 · 2015

Earlier work this paper cites.

Squad: 100,000+ questions for machine comprehension of text

Original

Pranav Rajpurkar, Jian Zhang, Konstantin Liu, and Percy Liang. 2016 · 2016

Earlier work this paper cites.

Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension

Original

Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017 · 2017

Earlier work this paper cites.

Outrageously large neural networks: The sparsely-gated mixture-of-experts layer

Original

Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017 · 2017

Earlier work this paper cites.

Mixed precision training

Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. 2018 · 2018

Earlier work this paper cites.

Improving language understanding by generative pre-training

Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018 · 2018

Earlier work this paper cites.

Adafactor: Adaptive learning rates with sublinear memory cost

Noam Shazeer and Mitchell Stern. 2018 · 2018

Earlier work this paper cites.

Bfloat16 processing for neural networks

Neil Burgess, Jelena Milanovic, Nigel Stephens, Konstantinos Monachopoulos, and David Mansell. 2019 · 2019

Earlier work this paper cites.

BERT: Pre-training of deep bidirectional transformers for language understanding

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 · 2019

Earlier work this paper cites.

Natural questions: a benchmark for question answering research

Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019 · 2019

Earlier work this paper cites.

Decoupled weight decay regularization

Ilya Loshchilov and Frank Hutter. 2019 · 2019

Earlier work this paper cites.

Are sixteen heads really better than one?

Paul Michel, Omer Levy, and Graham Neubig. 2019 · 2019

Earlier work this paper cites.

Language models are unsupervised multitask learners

Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019 · 2019

Earlier work this paper cites.

Coqa: A conversational question answering challenge

Siva Reddy, Danqi Chen, and Christopher D Manning. 2019 · 2019

Earlier work this paper cites.

Efficientnet: Rethinking model scaling for convolutional neural networks

Mingxing Tan and Quoc Le. 2019 · 2019

Earlier work this paper cites.

Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned

Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. 2019 · 2019

Earlier work this paper cites.

Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search

Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. 2019 · 2019

Earlier work this paper cites.

Language models are few-shot learners

Original

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020 · 2020

Earlier work this paper cites.

Realtoxicityprompts: Evaluating neural toxic degeneration in language models

Original

Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020 · 2020

Earlier work this paper cites.

Single path one-shot neural architecture search with uniform sampling

Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun. 2020 · 2020

Earlier work this paper cites.

Tinybert: Distilling bert for natural language understanding

Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020 · 2020

Earlier work this paper cites.

Unity: A general platform for intelligent agents

Original

Arthur Juliani, Vincent-Pierre Berges, Ervin Teng, Andrew Cohen, Jonathan Harper, Chris Elion, Chris Goy, Yuan Gao, Hunter Henry, Marwan Mattar, and Danny Lange. 2020 · 2020

Earlier work this paper cites.

Transformers are rnns: Fast autoregressive transformers with linear attention

Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. 2020 · 2020

Earlier work this paper cites.

Zero: Memory optimizations toward training trillion parameter models

Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020 · 2020

Earlier work this paper cites.

A primer in BERTology: What we know about how BERT works

Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020 · 2020

Earlier work this paper cites.

Superglue: Learning feature matching with graph neural networks

Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. 2020 · 2020

Earlier work this paper cites.

MobileBERT: a compact task-agnostic BERT for resource-limited devices

Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. 2020 · 2020

Earlier work this paper cites.

Towards debiasing NLU models from unknown biases

Prasetya Ajie Utama, Nafise Sadat Moosavi, and Iryna Gurevych. 2020 · 2020

Earlier work this paper cites.

Structured pruning of large language models

Ziheng Wang, Jeremy Wohlwend, and Tao Lei. 2020b · 2020

Earlier work this paper cites.

Improved analysis of clipping algorithms for non-convex optimization

Bohang Zhang, Jikai Jin, Cong Fang, and Liwei Wang. 2020 · 2020

Earlier work this paper cites.

Medically aware gpt-3 as a data generator for medical dialogue summarization

Bharath Chintagunta, Namit Katariya, Xavier Amatriain, and Anitha Kannan. 2021 · 2021

Earlier work this paper cites.

Supershaper: Task-agnostic super pre-training of bert models with variable hidden dimensions

Original

Vinod Ganesan, Gowtham Ramesh, and Pratyush Kumar. 2021 · 2021

Earlier work this paper cites.

Lora: Low-rank adaptation of large language models

Original

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021 · 2021

Earlier work this paper cites.

Mimic-iv

Alistair Johnson, Lucas Bulgarelli, Tom Pollard, Steven Horng, Leo Anthony Celi, and Roger Mark. 2020 · 2021

Earlier work this paper cites.

The power of scale for parameter-efficient prompt tuning

Original

Brian Lester, Rami Al-Rfou, and Noah Constant. 2021 · 2021

Earlier work this paper cites.

Recent advances in natural language processing via large pre-trained language models: A survey

Bonan Min, Hayley Ross, Elior Sulem, Amir Pouran Ben Veyseh, Thien Huu Nguyen, Oscar Sainz, Eneko Agirre, Ilana Heinz, and Dan Roth. 2021 · 2021

Earlier work this paper cites.

StereoSet: Measuring stereotypical bias in pretrained language models

Moin Nadeem, Anna Bethke, and Siva Reddy. 2021 · 2021

Earlier work this paper cites.

Scaling language models: Methods, analysis & insights from training gopher

Original

Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. 2021 · 2021

Cited alongside, same era.

Efficient content-based sparse attention with routing transformers

Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. 2021 · 2021

Cited alongside, same era.

Nyströmformer: A nyström-based algorithm for approximating self-attention

Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. 2021 · 2021

Cited alongside, same era.

Learning n: m fine-grained structured sparse neural networks from scratch

Original

Aojun Zhou, Yukun Ma, Junnan Zhu, Jianbo Liu, Zhijie Zhang, Kun Yuan, Wenxiu Sun, and Hongsheng Li. 2021 · 2021

Cited alongside, same era.

Neural architecture search with reinforcement learning

Paligemma: A versatile 3b vlm for transfer

Original

Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, et al. 2024 · 2024

Closest in time.

Towards cross-tokenizer distillation: the universal logit distillation loss for llms

Original

Nicolas Boizard, Kevin El Haddad, Céline Hudelot, and Pierre Colombo. 2024 · 2024

Closest in time.

Fast and optimal weight update for pruned large language models

Original

Vladimír Boža. 2024 · 2024

Closest in time.

Retaining key information under high compression ratios: Query-guided compressor for llms

Zhiwei Cao, Qian Cao, Yu Lu, Ningxin Peng, Luyang Huang, Shanbo Cheng, and Jinsong Su. 2024 · 2024

Closest in time.

A survey on evaluation of large language models

Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2024 · 2024