Fetching the paper…
Reading the bibliography…
Effectively scaling large Transformer models is a main driver of recent advances in natural language processing.
Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019 · 1910
Earlier work this paper cites.
Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. 2019 · 1910
Earlier work this paper cites.
Adaptive mixtures of local experts
Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. 1991 · 1991
Earlier work this paper cites.
Auction algorithms for network flow problems: A tutorial introduction
Dimitri P. Bertsekas. 1992 · 1992
Earlier work this paper cites.
Scaling laws for neural language models
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020 · 2001
Earlier work this paper cites.
Whale: Scaling deep learning model training to the trillions
Xianyan Jia, Le Jiang, Ang Wang, Jie Zhang, Xinyuan Li, Wencong Xiao, Yong Li, Zhen Zheng, Xiaoyong Liu, Wei Lin, et al. 2020 · 2011
Earlier work this paper cites.
Distilling the knowledge in a neural network
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. 2015 · 2015
Earlier work this paper cites.
Big/little deep neural network for ultra low power inference
Eunhyeok Park, Dongyoung Kim, Soobeom Kim, Yong-Deok Kim, Gunhee Kim, Sungroh Yoon, and Sungjoo Yoo. 2015 · 2015
Earlier work this paper cites.
Branchynet: Fast inference via early exiting from deep neural networks
Surat Teerapittayanon, Bradley McDanel, and H. T. Kung. 2016 · 2016
Earlier work this paper cites.
Hierarchical multiscale recurrent neural networks
Junyoung Chung, Sungjin Ahn, and Yoshua Bengio. 2017 · 2017
Earlier work this paper cites.
Variable computation in recurrent neural networks
Yacine Jernite, Edouard Grave, Armand Joulin, and Tomás Mikolov. 2017 · 2017
Earlier work this paper cites.
Outrageously large neural networks: The sparsely-gated mixture-of-experts layer
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E. Hinton, and Jeff Dean. 2017 · 2017
Earlier work this paper cites.
Reasonet: Learning to stop reading in machine comprehension
Yelong Shen, Po-Sen Huang, Jianfeng Gao, and Weizhu Chen. 2017 · 2017
Earlier work this paper cites.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017 · 2017
Earlier work this paper cites.
Learning to skim text
Adams Wei Yu, Hongrae Lee, and Quoc V. Le. 2017 · 2017
Earlier work this paper cites.
Skip RNN: learning to skip state updates in recurrent neural networks
Víctor Campos, Brendan Jou, Xavier Giró-i-Nieto, Jordi Torres, and Shih-Fu Chang. 2018 · 2018
Earlier work this paper cites.
Speed reading: Learning to read forbackward via shuttle
Tsu-Jui Fu and Wei-Yun Ma. 2018 · 2018
Earlier work this paper cites.
Focused hierarchical rnns for conditional sequence processing
Nan Rosemary Ke, Konrad Zolna, Alessandro Sordoni, Zhouhan Lin, Adam Trischler, Yoshua Bengio, Joelle Pineau, Laurent Charlin, and Christopher J. Pal. 2018 · 2018
Earlier work this paper cites.
Jumper: Learning when to make classification decisions in reading
Xianggen Liu, Lili Mou, Haotian Cui, Zhengdong Lu, and Sen Song. 2018 · 2018
Earlier work this paper cites.
Neural speed reading via skim-rnn
Min Joon Seo, Sewon Min, Ali Farhadi, and Hannaneh Hajishirzi. 2018 · 2018
Earlier work this paper cites.
Mesh-tensorflow: Deep learning for supercomputers
Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, Ryan Sepassi, and Blake A. Hechtman. 2018 · 2018
Earlier work this paper cites.
Fast and accurate text classification: Skimming, rereading and early stopping
Keyi Yu, Yang Liu, Alexander G. Schwing, and Jian Peng. 2018 · 2018
Earlier work this paper cites.
BERT: pre-training of deep bidirectional transformers for language understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 · 2019
Cited alongside, same era.
Neural speed reading with structural-jump-lstm
Christian Hansen, Casper Hansen, Stephen Alstrup, Jakob Grue Simonsen, and Christina Lioma. 2019 · 2019
Cited alongside, same era.
Shallow-deep networks: Understanding and mitigating network overthinking
Yigitcan Kaya, Sanghyun Hong, and Tudor Dumitras. 2019 · 2019
Cited alongside, same era.
Teach machine how to read: Reading behavior inspired relevance estimation
Xiangsheng Li, Jiaxin Mao, Chao Wang, Yiqun Liu, Min Zhang, and Shaoping Ma. 2019 · 2019
Cited alongside, same era.
Hardware-software co-design approach for deep learning inference
Debdeep Paul, Jawar Singh, and Jimson Mathew. 2019 · 2019
Cited alongside, same era.
Language models are few-shot learners
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020 · 2020
BASE layers: Simplifying training of large, sparse models
Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. 2021 · 2021
Later among the works it cites.
M6: A chinese multimodal pretrainer
Junyang Lin, Rui Men, An Yang, Chang Zhou, Ming Ding, Yichang Zhang, Peng Wang, Ang Wang, Le Jiang, Xianyan Jia, et al. 2021 · 2021
Later among the works it cites.
Dense-to-sparse gate for mixture-of-experts
Xiaonan Nie, Shijie Cao, Xupeng Miao, Lingxiao Ma, Jilong Xue, Youshan Miao, Zichao Yang, Zhi Yang, and Bin Cui. 2021 · 2021
Later among the works it cites.
Hash layers for large sparse models
Stephen Roller, Sainbayar Sukhbaatar, Jason Weston, et al. 2021 · 2021
Later among the works it cites.
Consistent accelerated inference via confident adaptive transformers
Tal Schuster, Adam Fisch, Tommi Jaakkola, and Regina Barzilay. 2021 · 2021
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Cited alongside, same era.
Depth-adaptive transformer
Maha Elbayad, Jiatao Gu, Edouard Grave, and Michael Auli. 2020 · 2020
Cited alongside, same era.
Power-bert: Accelerating BERT inference via progressive word-vector elimination
Saurabh Goyal, Anamitra Roy Choudhury, Saurabh Raje, et al. 2020 · 2020
Cited alongside, same era.
ALBERT: A lite BERT for self-supervised learning of language representations
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020 · 2020
Cited alongside, same era.
Fastbert: a self-distilling BERT with adaptive inference time
Weijie Liu, Peng Zhou, Zhiruo Wang, Zhe Zhao, Haotang Deng, and Qi Ju. 2020 · 2020
Cited alongside, same era.
Exploring the limits of transfer learning with a unified text-to-text transformer
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020 · 2020
Cited alongside, same era.
The right tool for the job: Matching model and instance complexities
Roy Schwartz, Gabriel Stanovsky, Swabha Swayamdipta, Jesse Dodge, and Noah A. Smith. 2020 · 2020
Cited alongside, same era.
Later among the works it cites.
Early exiting with ensemble internal classifiers
Tianxiang Sun, Yunhua Zhou, Xiangyang Liu, Xinyu Zhang, Hao Jiang, Zhao Cao, Xuanjing Huang, and Xipeng Qiu. 2021 · 2021
Later among the works it cites.
Berxit: Early exiting for BERT with better fine-tuning and extension to regression
Ji Xin, Raphael Tang, Yaoliang Yu, and Jimmy Lin. 2021 · 2021
Later among the works it cites.
Beyond preserved accuracy: Evaluating loyalty and robustness of BERT compression
Canwen Xu, Wangchunshu Zhou, Tao Ge, Ke Xu, Julian J. McAuley, and Furu Wei. 2021 · 2021
Later among the works it cites.
M6-t: Exploring sparse expert models and beyond
An Yang, Junyang Lin, Rui Men, Chang Zhou, Le Jiang, Xianyan Jia, Ang Wang, Jie Zhang, Jiamang Wang, Yong Li, et al. 2021 · 2021
Later among the works it cites.
TR-BERT: dynamic token reduction for accelerating BERT inference
Deming Ye, Yankai Lin, Yufei Huang, and Maosong Sun. 2021 · 2021
Later among the works it cites.
Moefication: Conditional computation of transformer models for efficient inference
Zhengyan Zhang, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2021 · 2021
Later among the works it cites.
Leebert: Learned early exit for BERT with cross-level optimization
Wei Zhu. 2021 · 2021
Later among the works it cites.
Unified scaling laws for routed language models
Aidan Clark, Diego de las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, et al. 2022 · 2022
Closest in time.
Transkimmer: Transformer learns to layer-wise skim
Yue Guan, Zhengyi Li, Jingwen Leng, et al. 2022 · 2022
Closest in time.
Learned token pruning for transformers
Sehoon Kim, Sheng Shen, David Thorsley, et al. 2022 · 2022
Closest in time.
Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation ai scale
Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, and Yuxiong He. 2022 · 2022
Closest in time.
Skipbert: Efficient inference with shallow layer skipping
Jue Wang, Ke Chen, Gang Chen, et al. 2022 · 2022
Closest in time.
A survey on model compression and acceleration for pretrained language models
Canwen Xu and Julian McAuley. 2022 · 2022
Closest in time.
Taming sparsely activated transformer with stochastic experts
Simiao Zuo, Xiaodong Liu, Jian Jiao, Young Jin Kim, Hany Hassan, Ruofei Zhang, Jianfeng Gao, and Tuo Zhao. 2022 · 2022
Closest in time.
A global past-future early exit method for accelerating inference of pre-trained language models
Kaiyuan Liao, Yi Zhang, Xuancheng Ren, Qi Su, Xu Sun, and Bin He. 2021 · 2023
Closest in time.
Show, attend and tell: Neural image caption generation with visual attention
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C. Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio. 2015 · 2057
Closest in time.