Optimized Speculative Sampling for GPU Hardware Accelerators

Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter

Original

Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019 · 1910

Earlier work this paper cites.

Fast transformer decoding: One write-head is all you need

Original

Noam Shazeer. 2019 · 1911

Earlier work this paper cites.

Training stochastic model recognition algorithms as networks can lead to maximum mutual information estimation of parameters

John Bridle. 1989 · 1989

Earlier work this paper cites.

The cache performance and optimizations of blocked algorithms

Monica D. Lam, Edward E. Rothberg, and Michael E. Wolf. 1991 · 1991

Earlier work this paper cites.

GLU variants improve transformer

Original

Noam Shazeer. 2020 · 2002

Earlier work this paper cites.

Slice sampling

Radford M. Neal. 2003 · 2003

Earlier work this paper cites.

ROUGE: A package for automatic evaluation of summaries

Chin-Yew Lin. 2004 · 2004

Earlier work this paper cites.

Program optimization space pruning for a multithreaded gpu

Shane Ryoo, Christopher I Rodrigues, Sam S Stone, Sara S Baghsorkhi, Sain-Zee Ueng, John A Stratton, and Wen-mei W Hwu. 2008 · 2008

Earlier work this paper cites.

TED-LIUM: an automatic speech recognition dedicated corpus

Anthony Rousseau, Paul Deléglise, and Yannick Estève. 2012 · 2012

Earlier work this paper cites.

Professional CUDA C Programming

J. Cheng, M. Grossman, and T. McKercher. 2014 · 2014

Earlier work this paper cites.

Distilling the knowledge in a neural network

Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. 2015 · 2015

Earlier work this paper cites.

Librispeech: An asr corpus based on public domain audio books

Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015 · 2015

Earlier work this paper cites.

Abstractive text summarization using sequence-to-sequence RNNs and beyond

Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Çağlar Gulçehre, and Bing Xiang. 2016 · 2016

Earlier work this paper cites.

Efficient softmax approximation for GPUs

Édouard Grave, Armand Joulin, Moustapha Cissé, David Grangier Facebook AI Research, and Hervé Jégou. 2017 · 2017

Earlier work this paper cites.

Svd-softmax: Fast softmax approximation on large vocabulary neural networks

Kyuhong Shim, Minjae Lee, Iksoo Choi, Yoonho Boo, and Wonyong Sung. 2017 · 2017

Earlier work this paper cites.

Attention is all you need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017 · 2017

Earlier work this paper cites.

Hardware-aware softmax approximation for deep neural networks

Xue Geng, Jie Lin, Bin Zhao, Anmin Kong, Mohamed M. Sabry Aly, and Vijay Ramaseshan Chandrasekhar. 2018 · 2018

Earlier work this paper cites.

Dissecting the NVIDIA volta GPU architecture via microbenchmarking

Original

Zhe Jia, Marco Maggioni, Benjamin Staiger, and Daniele P. Scarpazza. 2018 · 2018

Earlier work this paper cites.

Online normalizer calculation for softmax

Original

Maxim Milakov and Natalia Gimelshein. 2018 · 2018

Earlier work this paper cites.

Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization

Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018 · 2018

Earlier work this paper cites.

Blockwise parallel decoding for deep autoregressive models

Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. 2018 · 2018

Earlier work this paper cites.

Pytorch: An imperative style, high-performance deep learning library

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019 · 2019

Earlier work this paper cites.

Patient knowledge distillation for BERT model compression

Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. 2019 · 2019

Cited alongside, same era.

Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned

Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. 2019 · 2019

Cited alongside, same era.

Root mean square layer normalization

Biao Zhang and Rico Sennrich. 2019 · 2019

Cited alongside, same era.

Common Voice: A massively-multilingual speech corpus

Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gregor Weber. 2020 · 2020

Cited alongside, same era.

Infinite attention: Nngp and ntk for deep attention networks

Jiri Hron, Yasaman Bahri, Jascha Narain Sohl-Dickstein, and Roman Novak. 2020 · 2020

Cited alongside, same era.

TinyBERT: Distilling BERT for natural language understanding

Distil-whisper: Robust knowledge distillation via large-scale pseudo labelling

Original

Sanchit Gandhi, Patrick von Platen, and Alexander M Rush. 2023 · 2023

Later among the works it cites.

Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes

Cheng-Yu Hsieh, Chun-Liang Li, Chih-kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alex Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. 2023 · 2023

Later among the works it cites.

Softmax output approximation for activation memory-efficient training of attention-based networks

Changhyeon Lee and Seulki Lee. 2023 · 2023

Later among the works it cites.

Fast inference from transformers via speculative decoding

Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023 · 2023

Later among the works it cites.

PaSS: Parallel speculative sampling

Original

Giovanni Monea, Armand Joulin, and Edouard Grave. 2023 · 2023