MCSD: An Efficient Language Model with Diverse Fusion

Language models are few-shot learners

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020 · 1901

Earlier work this paper cites.

Root mean square layer normalization

Original

Biao Zhang and Rico Sennrich. 2019 · 1910

Earlier work this paper cites.

Fast transformer decoding: One write-head is all you need

Original

Noam Shazeer. 2019 · 1911

Earlier work this paper cites.

Glu variants improve transformer

Original

Noam Shazeer. 2020 · 2002

Earlier work this paper cites.

Language modeling with gated convolutional networks

Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. 2017 · 2017

Earlier work this paper cites.

Sigmoid-weighted linear units for neural network function approximation in reinforcement learning

Original

Stefan Elfwing, Eiji Uchibe, and Kenji Doya. 2017 · 2017

Earlier work this paper cites.

Attention is all you need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017 · 2017

Earlier work this paper cites.

Decoupled weight decay regularization

Original

Ilya Loshchilov and Frank Hutter. 2019 · 2019

Cited alongside, same era.

An attention free transformer

Original

Shuangfei Zhai, Walter Talbott, Nitish Srivastava, Chen Huang, Hanlin Goh, Ruixiang Zhang, and Josh Susskind. 2021 · 2021

Cited alongside, same era.

Training compute-optimal large language models

Original

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. 2022 · 2022

Cited alongside, same era.

Gqa: Training generalized multi-query transformer models from multi-head checkpoints

Original

Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. 2023 · 2023

Cited alongside, same era.

Rwkv: Reinventing rnns for the transformer era

Original

Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, Kranthi Kiran GV, Xuzheng He, Haowen Hou, Jiaju Lin, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartlomiej Koptyra, Hayden Lau, Krishna Sri Ipsit Mantri, Ferdinand Mom, Atsushi Saito, Guangyu Song, Xiangru Tang, Bolun Wang, Johan S. Wind, Stanislaw Wozniak, Ruichong Zhang, Zhenyuan Zhang, Qihang Zhao, Peng Zhou, Qinghua Zhou, Jian Zhu, and Rui-Jie Zhu. 2023 · 2023

Later among the works it cites.

Retentive network: A successor to transformer for large language models

Original

Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. 2023 · 2023

Later among the works it cites.

Llama: Open and efficient foundation language models

Original

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023 · 2023

Later among the works it cites.

Griffin: Mixing gated linear recurrences with local attention for efficient language models