GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints

Fast transformer decoding: One write-head is all you need

Original

Noam Shazeer. 2019 · 1911

Earlier work this paper cites.

Roofline: an insightful visual performance model for multicore architectures

Samuel Williams, Andrew Waterman, and David A. Patterson. 2009 · 2009

Earlier work this paper cites.

Distilling the knowledge in a neural network

Original

Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. 2015 · 2015

Earlier work this paper cites.

Abstractive text summarization using sequence-to-sequence rnns and beyond

Ramesh Nallapati, Bowen Zhou, Cícero Nogueira dos Santos, Çaglar Gülçehre, and Bing Xiang. 2016 · 2016

Earlier work this paper cites.

Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension

Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017 · 2017

Earlier work this paper cites.

JAX: composable transformations of Python+NumPy programs

James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. 2018 · 2018

Earlier work this paper cites.

A discourse-aware attention model for abstractive summarization of long documents

Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018 · 2018

Earlier work this paper cites.

Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model

Alexander R. Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir R. Radev. 2019 · 2019

Earlier work this paper cites.

GLUE: A multi-task benchmark and analysis platform for natural language understanding

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019 · 2019

Earlier work this paper cites.

Flax: A neural network library and ecosystem for JAX

Jonathan Heek, Anselm Levskaya, Avital Oliver, Marvin Ritter, Bertrand Rondepierre, Andreas Steiner, and Marc van Zee. 2020 · 2020

Earlier work this paper cites.

Scale down transformer by grouping features for a lightweight character-level language model

Sungrae Park, Geewook Kim, Junyeop Lee, Junbum Cha, Ji-Hoon Kim, and Hwalsuk Lee. 2020 · 2020

Cited alongside, same era.

Exploring the limits of transfer learning with a unified text-to-text transformer

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020 · 2020

Cited alongside, same era.

Knowledge distillation: A survey

Jianping Gou, Baosheng Yu, Stephen J. Maybank, and Dacheng Tao. 2021 · 2021

Cited alongside, same era.

Mediasum: A large-scale media interview dataset for dialogue summarization

Chenguang Zhu, Yang Liu, Jie Mei, and Michael Zeng. 2021 · 2021

Cited alongside, same era.

Palm: Scaling language modeling with pathways

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022 · 2022

Profile your model with cloud tpu tools

Google. 2020 · 2022

Later among the works it cites.

Sparse upcycling: Training mixture-of-experts from dense checkpoints

Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp, Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostafa Dehghani, and Neil Houlsby. 2022 · 2022

Later among the works it cites.

Fast inference from transformers via speculative decoding

Original

Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2022 · 2022

Later among the works it cites.

Towards lightweight transformer via group-wise transformation for vision-and-language tasks

Gen Luo, Yiyi Zhou, Xiaoshuai Sun, Yan Wang, Liujuan Cao, Yongjian Wu, Feiyue Huang, and Rongrong Ji. 2022 · 2022

Later among the works it cites.

Efficiently scaling transformer inference

Original

Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Anselm Levskaya, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. 2022 · 2022