Fetching the paper…
Reading the bibliography…
Scaling language models to larger and deeper sizes has led to significant boosts in performance.
Learning internal representations by error propagation, parallel distributed processing, explorations in the microstructure of cognition, ed. de rumelhart and j. mcclelland. vol. 1. 1986
David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams · 1986
Earlier work this paper cites.
Adaptive Computation Time for Recurrent Neural Networks, 2016
Alex Graves · 2016
Earlier work this paper cites.
Adaptive neural networks for efficient inference
Tolga Bolukbasi, Joseph Wang, Ofer Dekel, and Venkatesh Saligrama · 2017
Earlier work this paper cites.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin · 2017
Earlier work this paper cites.
Universal transformers
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser · 2019
Earlier work this paper cites.
Decoupled weight decay regularization
Ilya Loshchilov and Frank Hutter · 2019
Earlier work this paper cites.
Slimmable neural networks
Jiahui Yu, Linjie Yang, Ning Xu, Jianchao Yang, and Thomas S. Huang · 2019
Earlier work this paper cites.
Language models are few-shot learners
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei · 2020
Earlier work this paper cites.
Depth-adaptive transformer
Maha Elbayad, Jiatao Gu, Edouard Grave, and Michael Auli · 2020
Earlier work this paper cites.
Efficient Transformers: A Survey, 2020
Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler · 2020
Earlier work this paper cites.
On layer normalization in the transformer architecture
Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tie-Yan Liu · 2020
Earlier work this paper cites.
PonderNet: Learning to Ponder, 2021
Andrea Banino, Jan Balaguer, and Charles Blundell · 2021
Earlier work this paper cites.
Training Verifiers to Solve Math Word Problems, 2021
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman · 2021
Earlier work this paper cites.
Compressing large-scale transformer-based models: A case study on BERT
Prakhar Ganesh, Yao Chen, Xin Lou, Mohammad Ali Khan, Yin Yang, Hassan Sajjad, Preslav Nakov, Deming Chen, and Marianne Winslett · 2021
Cited alongside, same era.
OpenWebText2 dataset, as part of ‘the Pile: An 800gb dataset of diverse text for language modeling‘
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy · 2021
Cited alongside, same era.
Faster depth-adaptive transformers
Yijin Liu, Fandong Meng, Jie Zhou, Yufeng Chen, and Jinan Xu · 2021
Cited alongside, same era.
Faster depth-adaptive transformers
Yijin Liu, Fandong Meng, Jie Zhou, Yufeng Chen, and Jinan Xu · 2021
Cited alongside, same era.
Show Your Work: Scratchpads for Intermediate Computation with Language Models, 2021
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena · 2021
Cited alongside, same era.
Think before you speak: Training Language Models With Pause Tokens, 2023
Sachin Goyal, Ziwei Ji, Ankit Singh Rawat, Aditya Krishna Menon, Sanjiv Kumar, and Vaishnavh Nagarajan · 2023
Closest in time.
Mistral 7B, 2023
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed · 2023
Closest in time.
Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Step, 2023
Liunian Harold Li, Jack Hessel, Youngjae Yu, Xiang Ren, Kai-Wei Chang, and Yejin Choi · 2023
Closest in time.
Auto-Regressive Next-Token Predictors are Universal Learners, 2023
Eran Malach · 2023
Closest in time.
The Expressive Power of Transformers with Chain of Thought, 2023
William Merrill and Ashish Sabharwal · 2023
Closest in time.
GPT-4 Technical Report, 2023
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Lessons on parameter sharing across layers in transformers
Sho Takase and Shun Kiyono · 2021
Cited alongside, same era.
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, 2022
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré · 2022
Cited alongside, same era.
Large Language Models Are Reasoning Teachers, 2022
Namgyu Ho, Laura Schmid, and Se-Young Yun · 2022
Cited alongside, same era.
Large language models are zero-shot reasoners
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa · 2022
Cited alongside, same era.
The Parallelism Tradeoff: Limitations of Log-Precision Transformers, 2022
William Merrill and Ashish Sabharwal · 2022
Cited alongside, same era.
Masked training of neural networks with partial gradients
Amirkeivan Mohtashami, Martin Jaggi, and Sebastian U. Stich · 2022
Cited alongside, same era.
Foundation Transformers, 2022
Hongyu Wang, Shuming Ma, Shaohan Huang, Li Dong, Wenhui Wang, Zhiliang Peng, Yu Wu, Payal Bajaj, Saksham Singhal, Alon Benhaim, Barun Patra, Zhun Liu, Vishrav Chaudhary, Xia Song, and Furu Wei · 2022
Cited alongside, same era.
OpenAI · 2023
Closest in time.
Faster Causal Attention Over Large Sequences Through Sparse Flash Attention, 2023
Matteo Pagliardini, Daniele Paliotta, Martin Jaggi, and François Fleuret · 2023
Closest in time.
Sparse Universal Transformer, 2023
Shawn Tan, Yikang Shen, Zhenfang Chen, Aaron Courville, and Chuang Gan · 2023
Closest in time.
Mixtral of Experts, 2024
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed · 2024
Closest in time.
DenseFormer: Enhancing Information Flow in Transformers via Depth Weighted Averaging, 2024
Matteo Pagliardini, Amirkeivan Mohtashami, Francois Fleuret, and Martin Jaggi · 2024
Closest in time.
Mixture-of-Depths: Dynamically allocating compute in transformer-based language models, 2024
David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, and Adam Santoro · 2024
Closest in time.
Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking, 2024
Eric Zelikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah D. Goodman · 2024
Closest in time.