Fetching the paper…
Reading the bibliography…
Convolution models with long filters have demonstrated state-of-the-art reasoning abilities in many long-sequence tasks but lag behind the most optimized Transformers in wall-clock time.
An algorithm for the machine calculation of complex fourier series
James W Cooley and John W Tukey · 1965
Earlier work this paper cites.
Applications of digital signal processing
Alan V Oppenheim · 1978
Earlier work this paper cites.
Displacement ranks of matrices and linear equations
Thomas Kailath, Sun-Yuan Kung, and Martin Morf · 1979
Earlier work this paper cites.
Real-valued fast fourier transform algorithms
H V Sorensen, D Jones, Michael Heideman, and C Burrus · 1987
Earlier work this paper cites.
The fast Fourier transform and its applications
E Oran Brigham · 1988
Earlier work this paper cites.
Ffts in external of hierarchical memory
David H Bailey · 1989
Earlier work this paper cites.
Inside the FFT black box: serial and parallel fast Fourier transform algorithms
Eleanor Chu and Alan George · 1999
Earlier work this paper cites.
On a new class of structured matrices
Yuli Eidelman and Israel Gohberg · 1999
Earlier work this paper cites.
Discrete-time signal processing. Vol. 2
Alan V Oppenheim, John R Buck, and Ronald W Schafer · 2001
Earlier work this paper cites.
Parallel fft algorithms on network-on-chips
Jun Ho Bahn, Jung Sook Yang, Wen-Hsiang Hu, and Nader Bagherzadeh · 2009
Earlier work this paper cites.
Pipelined parallel fft architectures via folding transformation
Manohar Ayinala, Michael Brown, and Keshab K Parhi · 2011
Earlier work this paper cites.
Computer architecture: a quantitative approach
John L Hennessy and David A Patterson · 2011
Earlier work this paper cites.
Imagenet classification with deep convolutional neural networks
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton · 2012
Earlier work this paper cites.
Song Han, Huizi Mao, and William J Dally · 2015
Earlier work this paper cites.
Learning both weights and connections for efficient neural network
Song Han, Jeff Pool, John Tran, and William Dally · 2015
Earlier work this paper cites.
Structured transforms for small-footprint deep learning
Vikas Sindhwani, Tara Sainath, and Sanjiv Kumar · 2015
Earlier work this paper cites.
Cooley-tukey fft algorithms
AJAA Bekele · 2016
Earlier work this paper cites.
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2016
Earlier work this paper cites.
Wavenet: A generative model for raw audio
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu · 2016
Earlier work this paper cites.
Invariance and stability of deep convolutional representations
Alberto Bietti and Julien Mairal · 2017
Earlier work this paper cites.
Learning to prune deep neural networks via layer-wise optimal brain surgeon
Xin Dong, Shangyu Chen, and Sinno Pan · 2017
Earlier work this paper cites.
Runtime neural pruning
Ji Lin, Yongming Rao, Jiwen Lu, and Jie Zhou · 2017
Earlier work this paper cites.
Nvidia Tesla V100 GPU architecture, 2017
NVIDIA · 2017
Earlier work this paper cites.
Neural discrete representation learning
Aaron Van Den Oord, Oriol Vinyals, et al · 2017
Earlier work this paper cites.
A two-pronged progress in structured dense matrix vector multiplication
Christopher De Sa, Albert Gu, Rohan Puttagunta, Christopher Ré, and Atri Rudra · 2018
Earlier work this paper cites.
The challenge of realistic music generation: modelling raw audio at scale
Sander Dieleman, Aaron van den Oord, and Karen Simonyan · 2018
Earlier work this paper cites.
The lottery ticket hypothesis: Finding sparse, trainable neural networks
Jonathan Frankle and Michael Carbin · 2018
Earlier work this paper cites.
Efficient neural audio synthesis
Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Edward Lockhart, Florian Stimberg, Aaron Oord, Sander Dieleman, and Koray Kavukcuoglu · 2018
Earlier work this paper cites.
Online normalizer calculation for softmax
Maxim Milakov and Natalia Gimelshein · 2018
Earlier work this paper cites.
Natural tts synthesis by conditioning wavenet on mel spectrogram predictions
Jonathan Shen, Ruoming Pang, Ron J Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, et al · 2018
Earlier work this paper cites.
High fidelity speech synthesis with adversarial networks
Mikołaj Bińkowski, Jeff Donahue, Sander Dieleman, Aidan Clark, Erich Elsen, Norman Casagrande, Luis C Cobo, and Karen Simonyan · 2019
Earlier work this paper cites.
Sparse networks from scratch: Faster training without losing performance
Tim Dettmers and Luke Zettlemoyer · 2019
Earlier work this paper cites.
Stabilizing the lottery ticket hypothesis
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin · 2019
Earlier work this paper cites.
Melgan: Generative adversarial networks for conditional waveform synthesis
Kundan Kumar, Rithesh Kumar, Thibault De Boissiere, Lucas Gestin, Wei Zhen Teoh, Jose Sotelo, Alexandre De Brebisson, Yoshua Bengio, and Aaron C Courville · 2019
Earlier work this paper cites.
A graph theoretic framework of recomputation algorithms for memory-efficient backpropagation
Mitsuru Kusumoto, Takuya Inoue, Gentaro Watanabe, Takuya Akiba, and Masanori Koyama · 2019
Earlier work this paper cites.
Pytorch: An imperative style, high-performance deep learning library
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al · 2019
Earlier work this paper cites.
Waveglow: A flow-based generative network for speech synthesis
Ryan Prenger, Rafael Valle, and Bryan Catanzaro · 2019
Earlier work this paper cites.
Longformer: The long-document transformer
Iz Beltagy, Matthew E Peters, and Arman Cohan · 2020
Earlier work this paper cites.
Language models are few-shot learners
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al · 2020
Earlier work this paper cites.
Rethinking attention with performers
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al · 2020
Cited alongside, same era.
Funnel-transformer: Filtering out sequential redundancy for efficient language processing
Zihang Dai, Guokun Lai, Yiming Yang, and Quoc Le · 2020
Cited alongside, same era.
Learning fast algorithms for linear transforms using butterfly factorizations
Tri Dao, Albert Gu, Matthew Eichhorn, Atri Rudra, and Christopher Ré · 2020
Cited alongside, same era.
Jukebox: A generative model for music
Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever · 2020
Cited alongside, same era.
Linear mode connectivity and the lottery ticket hypothesis
How to train your hippo: State space models with generalized orthogonal basis projections
Albert Gu, Isys Johnson, Aman Timalsina, Atri Rudra, and Christopher Ré · 2022
Later among the works it cites.
Diagonal state spaces are as effective as structured state spaces
Ankit Gupta, Albert Gu, and Jonathan Berant · 2022
Later among the works it cites.
Liquid structural state-space models
Ramin Hasani, Mathias Lechner, Tsun-Huang Wang, Makram Chahine, Alexander Amini, and Daniela Rus · 2022
Later among the works it cites.
Powering extreme-scale hpc with cerebras wafer-scale accelerators
Adam Lavely · 2022
Later among the works it cites.
What makes convolutional models great on long sequence modeling?
Yuhong Li, Tianle Cai, Yi Zhang, Deming Chen, and Debadeepta Dey · 2022
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin · 2020
Cited alongside, same era.
Transformers are rnns: Fast autoregressive transformers with linear attention
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret · 2020
Cited alongside, same era.
Reformer: The efficient transformer
Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya · 2020
Cited alongside, same era.
Nvidia A100 tensor core GPU architecture, 2020
NVIDIA · 2020
Cited alongside, same era.
Movement pruning: Adaptive sparsity by fine-tuning
Victor Sanh, Thomas Wolf, and Alexander Rush · 2020
Cited alongside, same era.
Long range arena: A benchmark for efficient transformers
Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler · 2020
Cited alongside, same era.
Linformer: Self-attention with linear complexity
Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma · 2020
Cited alongside, same era.
Big bird: Transformers for longer sequences
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al · 2020
Cited alongside, same era.
Later among the works it cites.
A convnet for the 2020s
Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie · 2022
Later among the works it cites.
Mega: moving average equipped gated attention
Xuezhe Ma, Chunting Zhou, Xiang Kong, Junxian He, Liangke Gui, Graham Neubig, Jonathan May, and Luke Zettlemoyer · 2022
Later among the works it cites.
Long range language modeling via gated state spaces
Harsh Mehta, Ankit Gupta, Ashok Cutkosky, and Behnam Neyshabur · 2022
Later among the works it cites.
S4nd: Modeling images and videos as multidimensional signals with state spaces
Eric Nguyen, Karan Goel, Albert Gu, Gordon Downs, Preey Shah, Tri Dao, Stephen Baccus, and Christopher Ré · 2022
Later among the works it cites.
Nvidia H100 tensor core GPU architecture, 2022
NVIDIA · 2022
Later among the works it cites.
Siyi Tang, Jared A Dunnmon, Liangqiong Qu, Khaled K Saab, Christopher Lee-Messer, and Daniel L Rubin · 2022
Later among the works it cites.
Efficient transformers: A survey
Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler · 2022
Later among the works it cites.
Junxiong Wang, Jing Nathan Yan, Albert Gu, and Alexander M Rush · 2022
Later among the works it cites.
Melon: Breaking the memory wall for resource-efficient on-device machine learning
Qipeng Wang, Mengwei Xu, Chao Jin, Xinran Dong, Jinliang Yuan, Xin Jin, Gang Huang, Yunxin Liu, and Xuanzhe Liu · 2022
Later among the works it cites.
A full-stack search technique for domain optimized deep learning accelerators
Dan Zhang, Safeen Huda, Ebrahim Songhori, Kartik Prabhu, Quoc Le, Anna Goldie, and Azalia Mirhoseini · 2022
Later among the works it cites.
Effectively modeling time series with simple discrete state spaces
Michael Zhang, Khaled Kamal Saab, Michael Poli, Tri Dao, Karan Goel, and Christopher Re · 2022
Later among the works it cites.
On io-efficient attention mechanisms: Context-aware bifurcated attention and the generalized multi-group attention
Ben Athiwaratkun, Sujan Kumar Gonugondla, Sanjay Krishna Gouda, Haifeng Qian, Hantian Ding, Qing Sun, Jun Wang, Liangfu Chen, Jiacheng Guo, Parminder Bhatia, et al · 2023
Closest in time.
Flashattention-2: Faster attention with better parallelism and work partitioning
Tri Dao · 2023
Closest in time.
Multi-head state space model for speech recognition
Yassir Fathullah, Chunyang Wu, Yuan Shangguan, Junteng Jia, Wenhan Xiong, Jay Mahadeokar, Chunxi Liu, Yangyang Shi, Ozlem Kalinli, Mike Seltzer, et al · 2023
Closest in time.
Monarch Mixer: A simple sub-quadratic GEMM-based architecture
Daniel Y. Fu, Simran Arora, Jessica Grogan, Isys Johnson, Sabri Eyuboglu, Armin W. Thomas, Benjamin F. Spector, Michael Poli, Atri Rudra, and Christopher Ré · 2023
Closest in time.
Hungry Hungry Hippos: Towards language modeling with state space models
Daniel Y. Fu, Tri Dao, Khaled K. Saab, Armin W. Thomas, Atri Rudra, and Christopher Ré · 2023
Closest in time.
Simple hardware-efficient long convolutions for sequence modeling
Daniel Y. Fu, Elliot L. Epstein, Eric Nguyen, Armin W. Thomas, Michael Zhang, Tri Dao, Atri Rudra, and Christopher Ré · 2023
Closest in time.
Efficient movie scene detection using state-space transformers
Md Mohaiminul Islam, Mahmudul Hasan, Kishan Shamsundar Athrey, Tony Braskich, and Gedas Bertasius · 2023
Closest in time.
Tpu v4: An optically reconfigurable supercomputer for machine learning with hardware support for embeddings
Norm Jouppi, George Kurian, Sheng Li, Peter Ma, Rahul Nagarajan, Lifeng Nai, Nishant Patil, Suvinay Subramanian, Andy Swing, Brian Towles, et al · 2023
Closest in time.
Smpconv: Self-moving point representations for continuous convolution
Sanghyeon Kim and Eunbyung Park · 2023
Closest in time.
Efficient memory management for large language model serving with pagedattention
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica · 2023
Closest in time.
Starcoder: may the source be with you!
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al · 2023
Closest in time.
Blockwise parallel transformer for long context large models
Hao Liu and Pieter Abbeel · 2023
Closest in time.
Towards quantitative precision for ecg analysis: Leveraging state space models, self-supervision and patient metadata
Temesgen Mehari and Nils Strodthoff · 2023
Closest in time.
Structured state space decoder for speech recognition and synthesis
Koichi Miyazaki, Masato Murata, and Tomoki Koriyama · 2023
Closest in time.
Hyenadna: Long-range genomic sequence modeling at single nucleotide resolution
Eric Nguyen, Michael Poli, Marjan Faizi, Armin Thomas, Callum Birch-Sykes, Michael Wornow, Aman Patel, Clayton Rabideau, Stefano Massaroli, Yoshua Bengio, et al · 2023
Closest in time.
Cuda c++ programming guide, 2023
NVIDIA · 2023
Closest in time.
cufftdx v1.1.0 documentation, 2023
NVIDIA · 2023
Closest in time.
Cutlass 3.2, 2023
NVIDIA · 2023
Closest in time.
Fast causal attention with dynamic sparsity
Daniele Paliotta, Matteo Pagliardini, Martin Jaggi, and François Fleuret · 2023
Closest in time.
Hyena hierarchy: Towards larger convolutional language models
Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré · 2023
Closest in time.
Simplified state space layers for sequence modeling
Jimmy TH Smith, Andrew Warrington, and Scott Linderman · 2023
Closest in time.
Selective structured state-spaces for long-form video understanding
Jue Wang, Wentao Zhu, Pichao Wang, Xiang Yu, Linda Liu, Mohamed Omar, and Raffay Hamid · 2023
Closest in time.