Cross-layer Attention Sharing for Pre-trained Large Language Models

ZeRO: Memory Optimization Towards Training A Trillion Parameter Models

Original

Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2019 · 1910

Earlier work this paper cites.

Fast transformer decoding: One write-head is all you need

Original

Noam Shazeer. 2019 · 1911

Earlier work this paper cites.

NAS-BERT: Task-Agnostic and Adaptive-Size BERT Compression with Neural Architecture Search

Jin Xu, Xu Tan, Renqian Luo, Kaitao Song, Jian Li, Tao Qin, and Tie-Yan Liu. 2021 · 1943

Earlier work this paper cites.

Robust estimation of a location parameter

Peter J Huber. 1992 · 1992

Earlier work this paper cites.

Linformer: Self-Attention with Linear Complexity

Original

Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. 2020b · 2006

Earlier work this paper cites.

Effective Approaches to Attention-based Neural Machine Translation

Thang Luong, Hieu Pham, and Christopher D. Manning. 2015 · 2015

Earlier work this paper cites.

FitNets: Hints for Thin Deep Nets

Original

Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. 2015 · 2015

Earlier work this paper cites.

The LAMBADA dataset: Word prediction requiring a broad discourse context

Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. 2016 · 2016

Earlier work this paper cites.

BranchyNet: Fast inference via early exiting from deep neural networks

Surat Teerapittayanon, Bradley McDanel, and H. T. Kung. 2016 · 2016

Earlier work this paper cites.

The Reversible Residual Network: Backpropagation Without Storing Activations

Aidan N. Gomez, Mengye Ren, Raquel Urtasun, and Roger B. Grosse. 2017 · 2017

Earlier work this paper cites.

TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension

Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017 · 2017

Earlier work this paper cites.

Paraphrasing Complex Network: Network Compression via Factor Transfer

Jangho Kim, Seonguk Park, and Nojun Kwak. 2018 · 2018

Earlier work this paper cites.

Generating Wikipedia by Summarizing Long Sequences

Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018 · 2018

Earlier work this paper cites.

Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering

Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018 · 2018

Earlier work this paper cites.

Image Transformer

Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. 2018 · 2018

Earlier work this paper cites.

Learning Deep Representations with Probabilistic Knowledge Transfer

Nikolaos Passalis and Anastasios Tefas. 2018 · 2018

Earlier work this paper cites.

Self-Attentional Acoustic Models

Matthias Sperber, Jan Niehues, Graham Neubig, Sebastian Stüker, and Alex Waibel. 2018 · 2018

Earlier work this paper cites.

Accelerating Neural Transformer via an Average Attention Network

Biao Zhang, Deyi Xiong, and Jinsong Su. 2018 · 2018

Earlier work this paper cites.

BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions

Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019 · 2019

Earlier work this paper cites.

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 · 2019

Earlier work this paper cites.

Natural Questions: A Benchmark for Question Answering Research

Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019 · 2019

Earlier work this paper cites.

Are Sixteen Heads Really Better than One?

Paul Michel, Omer Levy, and Graham Neubig. 2019 · 2019

Earlier work this paper cites.

Language models are unsupervised multitask learners

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019 · 2019

Earlier work this paper cites.

CoQA: A Conversational Question Answering Challenge

Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019 · 2019

Earlier work this paper cites.

Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned

Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. 2019 · 2019

Earlier work this paper cites.

Sharing Attention Weights for Fast Transformer

Tong Xiao, Yinqiao Li, Jingbo Zhu, Zhengtao Yu, and Tongran Liu. 2019 · 2019

Earlier work this paper cites.

HellaSwag: Can a Machine Really Finish Your Sentence?

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019 · 2019

Earlier work this paper cites.

ETC: Encoding Long and Structured Inputs in Transformers

Joshua Ainslie, Santiago Ontañón, Chris Alberti, Vaclav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, and Li Yang. 2020 · 2020

Earlier work this paper cites.

PIQA: Reasoning about Physical Commonsense in Natural Language

Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020 · 2020

Earlier work this paper cites.

Reducing Transformer Depth on Demand with Structured Dropout

Angela Fan, Edouard Grave, and Armand Joulin. 2020 · 2020

Earlier work this paper cites.

Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learning

Mitchell A. Gordon, Kevin Duh, and Nicholas Andrews. 2020 · 2020

Earlier work this paper cites.

A 3 {}^{\mbox{3}} : Accelerating Attention Mechanisms in Neural Networks with Approximation

Tae Jun Ham, Sungjun Jung, Seonghak Kim, Young H. Oh, Yeonhong Park, Yoonho Song, Jung-Hun Park, Sanghee Lee, Kyoung Park, Jae W. Lee, and Deog-Kyoon Jeong. 2020 · 2020

Earlier work this paper cites.

ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Context

Wei Han, Zhengdong Zhang, Yu Zhang, Jiahui Yu, Chung-Cheng Chiu, James Qin, Anmol Gulati, Ruoming Pang, and Yonghui Wu. 2020 · 2020

Earlier work this paper cites.

TinyBERT: Distilling BERT for Natural Language Understanding

Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020 · 2020

Earlier work this paper cites.

Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention

Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. 2020 · 2020

Cited alongside, same era.

Reformer: The Efficient Transformer

Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020 · 2020

Cited alongside, same era.

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020 · 2020

Cited alongside, same era.

LadaBERT: Lightweight Adaptation of BERT through Hybrid Model Compression

Yihuan Mao, Yujing Wang, Chufan Wu, Chen Zhang, Yang Wang, Quanlu Zhang, Yaming Yang, Yunhai Tong, and Jing Bai. 2020 · 2020

Cited alongside, same era.

Movement Pruning: Adaptive Sparsity by Fine-Tuning

Victor Sanh, Thomas Wolf, and Alexander M. Rush. 2020 · 2020

Cited alongside, same era.

Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT

Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time

Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Ré, and Beidi Chen. 2023b · 2023

Later among the works it cites.

Split Computing and Early Exiting for Deep Learning Applications: Survey and Research Challenges

Yoshitomo Matsubara, Marco Levorato, and Francesco Restuccia. 2023 · 2023

Later among the works it cites.

One Wide Feedforward Is All You Need

Telmo Pires, António Vilarinho Lopes, Yannick Assogba, and Hendra Setiawan. 2023 · 2023

Later among the works it cites.

Stanford alpaca: An instruction-following llama model

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023 · 2023

Later among the works it cites.

Efficient Transformers: A Survey

Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. 2023 · 2023

Later among the works it cites.