Fetching the paper…
Reading the bibliography…
The challenge of mapping AI architectures to GPU hardware is creating a critical bottleneck in AI progress.
Pytorch: An imperative style, high-performance deep learning library, 2019
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala · 1912
Earlier work this paper cites.
An algorithm for the machine calculation of complex fourier series
James W Cooley and John W Tukey · 1965
Earlier work this paper cites.
Co-operating sequential processes
E. W. Dijkstra · 1966
Earlier work this paper cites.
Transformers are rnns: Fast autoregressive transformers with linear attention
A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret · 2006
Earlier work this paper cites.
Cudadma: Optimizing gpu memory bandwidth via warp specialization
Michael Bauer, Henry Cook, and Brucek Khailany · 2011
Earlier work this paper cites.
Halide: a language and compiler for optimizing parallelism, locality, and recomputation in image processing pipelines
Jonathan Ragan-Kelley, Connelly Barnes, Andrew Adams, Sylvain Paris, Frédo Durand, and Saman Amarasinghe · 2013
Earlier work this paper cites.
Tensorflow: Large-scale machine learning on heterogeneous distributed systems
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, et al · 2016
Earlier work this paper cites.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton · 2016
Earlier work this paper cites.
Cuda templates for linear algebra subroutines, 2017
NVIDIA · 2017
Earlier work this paper cites.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin · 2017
Earlier work this paper cites.
Dlvm: A modern compiler infrastructure for deep learning systems
Richard Wei, Lane Schwartz, and Vikram Adve · 2017
Earlier work this paper cites.
{ \{ TVM } \} : An automated { \{ End-to-End } \} optimizing compiler for deep learning
Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Haichen Shen, Meghan Cowan, Leyuan Wang, Yuwei Hu, Luis Ceze, et al · 2018
Earlier work this paper cites.
Diesel: Dsl for linear algebra and neural net computations on gpus
Venmugil Elango, Norm Rubin, Mahesh Ravishankar, Hariharan Sandanagobalane, and Vinod Grover · 2018
Earlier work this paper cites.
Glow: Graph lowering compiler techniques for neural networks
Nadav Rotem, Jordan Fix, Saleem Abdulrasool, Garret Catron, Summer Deng, Roman Dzhabarov, Nick Gibson, James Hegeman, Meghan Lele, Roman Levenstein, et al · 2018
Earlier work this paper cites.
Tensor comprehensions: Framework-agnostic high-performance machine learning abstractions
Nicolas Vasilache, Oleksandr Zinenko, Theodoros Theodoridis, Priya Goyal, Zachary DeVito, William S Moses, Sven Verdoolaege, Andrew Adams, and Albert Cohen · 2018
Cited alongside, same era.
Triton: an intermediate language and compiler for tiled neural network computations
Philippe Tillet, H. T. Kung, and David Cox · 2019
Cited alongside, same era.
Denoising diffusion probabilistic models, 2020
Jonathan Ho, Ajay Jain, and Pieter Abbeel · 2020
Cited alongside, same era.
Fast transformers with clustered attention
A. Vyas, A. Katharopoulos, and F. Fleuret · 2020
Cited alongside, same era.
Efficiently modeling long sequences with structured state spaces
Albert Gu, Karan Goel, and Christopher Ré · 2021
Cited alongside, same era.
Rwkv: Reinventing rnns for the transformer era
Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, Kranthi Kiran GV, Xuzheng He, Haowen Hou, Przemyslaw Kazienko, Jan Kocon, and Jiaming et al. Kong · 2023
Later among the works it cites.
Hyena hierarchy: Towards larger convolutional language models
Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré · 2023
Later among the works it cites.
Roformer: Enhanced transformer with rotary position embedding, 2023
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu · 2023
Later among the works it cites.
Spectral state space models, 2024
Naman Agarwal, Daniel Suo, Xinyi Chen, and Elad Hazan · 2024
Closest in time.
Linear transformers with learnable kernel functions are better in-context models
Yaroslav Aksenov, Nikita Balagansky, Sofia Maria Lo Cicero Vaina, Boris Shaposhnikov, Alexey Gorbatovski, and Daniil Gavrilov · 2024
Closest in time.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Ramin Hasani, Mathias Lechner, Tsun-Huang Wang, Makram Chahine, Alexander Amini, and Daniela Rus · 2022
Cited alongside, same era.
Gqa: Training generalized multi-query transformer models from multi-head checkpoints
Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai · 2023
Cited alongside, same era.
A case study in cuda kernel fusion: Implementing flashattention-2 on nvidia hopper architecture using the cutlass library, 2023a
Ganesh Bikshandi and Jay Shah · 2023
Cited alongside, same era.
Developing cuda kernels for accelerated matrix multiplication on nvidia hopper architecture using the cutlass library, 2023b
Ganesh Bikshandi and Jay Shah · 2023
Cited alongside, same era.
Monarch mixer: A simple sub-quadratic gemm-based architecture
Daniel Y. Fu, Simran Arora, Jessica Grogan, Isys Johnson, Sabri Eyuboglu, Armin W. Thomas, Benjamin Spector, Michael Poli, Atri Rudra, and Christopher Ré · 2023
Cited alongside, same era.
Mamba: Linear-time sequence modeling with selective state spaces
Albert Gu and Tri Dao · 2023
Cited alongside, same era.
On the computational complexity of self-attention
Feyza Duman Keles, Pruthuvi Mahesakya Wijewardena, and Chinmay Hegde · 2023
Cited alongside, same era.
Simple linear attention language models balance the recall-throughput tradeoff
Simran Arora, Sabri Eyuboglu, Michael Zhang, Aman Timalsina, Silas Alberti, Dylan Zinsley, James Zou, Atri Rudra, and Christopher Ré · 2024
Closest in time.
FlashAttention-2: Faster attention with better parallelism and work partitioning
Tri Dao · 2024
Closest in time.
Transformers are ssms: Generalized models and efficient algorithms through structured state space duality
Tri Dao and Albert Gu · 2024
Closest in time.
Flexattention: The flexibility of pytorch with the performance of flashattention, 2024
Horace He, Driss Guessous, Yanbo Liang, and Joy Dong · 2024
Closest in time.
Polysketchformer: Fast transformers via sketching polynomial kernels, 2024
Praneeth Kacham, Vahab Mirrokni, and Peilin Zhong · 2024
Closest in time.
Reverse-engineering cublas, 2024
Fabian Schuetze · 2024
Closest in time.
Flashattention-3: Fast and accurate attention with asynchrony and low-precision, 2024
Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao · 2024
Closest in time.
Fla: A triton-based library for hardware-efficient implementations of linear attention mechanism, January 2024
Songlin Yang and Yu Zhang · 2024
Closest in time.