Built on

Generating long sequences with sparse transformers
Original
R. Child, S. Gray, A. Radford, and I. Sutskever · 1904
Earlier work this paper cites.
Adaptively sparse transformers
Original
G. M. Correia, V. Niculae, and A. F. T. Martins · 1909
Earlier work this paper cites.
Blockwise self-attention for long document understanding
Original
J. Qiu, H. Ma, O. Levy, S. W. Yih, S. Wang, and J. Tang · 1911
Earlier work this paper cites.
Pytorch: An imperative style, high-performance deep learning library
Original
A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Z. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala · 1912
Earlier work this paper cites.
Reformer: The efficient transformer
Original
N. Kitaev, L. Kaiser, and A. Levskaya · 2001
Earlier work this paper cites.
Sparse sinkhorn attention
Original
Y. Tay, D. Bahri, L. Yang, D. Metzler, and D. Juan · 2002
Earlier work this paper cites.
Efficient content-based sparse attention with routing transformers
Original
A. Roy, M. Saffar, A. Vaswani, and D. Grangier · 2003
Earlier work this paper cites.
Longformer: The long-document transformer
Original
I. Beltagy, M. E. Peters, and A. Cohan · 2004
Earlier work this paper cites.
Language models are few-shot learners
Original
T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei · 2005
Earlier work this paper cites.
Synthesizer: Rethinking self-attention in transformer models
Original
Y. Tay, D. Bahri, D. Metzler, D. Juan, Z. Zhao, and C. Zheng · 2005
Earlier work this paper cites.
Transformers are rnns: Fast autoregressive transformers with linear attention
Original
A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret · 2006
Earlier work this paper cites.
Linformer: Self-attention with linear complexity
Original
S. Wang, B. Z. Li, M. Khabsa, H. Fang, and H. Ma · 2006
Earlier work this paper cites.

Similar

Random features for large-scale kernel machines
A. Rahimi and B. Recht · 2007
Cited alongside, same era.
Big bird: Transformers for longer sequences
Original
M. Zaheer, G. Guruganesh, A. Dubey, J. Ainslie, C. Alberti, S. Ontañón, P. Pham, A. Ravula, Q. Wang, L. Yang, and A. Ahmed · 2007
Cited alongside, same era.
Efficient transformers: A survey
Original
Y. Tay, M. Dehghani, D. Bahri, and D. Metzler · 2009
Cited alongside, same era.
Aligning books and movies: Towards story-like visual explanations by watching movies and reading books
Original
Y. Zhu, R. Kiros, R. Zemel, R. Salakhutdinov, R. Urtasun, A. Torralba, and S. Fidler · 2015
Cited alongside, same era.
Pointer sentinel mixture models
Original
S. Merity, C. Xiong, J. Bradbury, and R. Socher · 2016

Then

Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity
Original
W. Fedus, B. Zoph, and N. Shazeer · 2021
Later among the works it cites.
Random feature attention
H. Peng, N. Pappas, D. Yogatama, R. Schwartz, N. Smith, and L. Kong · 2021
Later among the works it cites.
Nyströmformer: A nyström-based algorithm for approximating self-attention
Original
Y. Xiong, Z. Zeng, R. Chakraborty, M. Tan, G. Fung, Y. Li, and V. Singh · 2021
Later among the works it cites.
Long-short transformer: Efficient transformers for language and vision
Original
C. Zhu, W. Ping, C. Xiao, M. Shoeybi, T. Goldstein, A. Anandkumar, and B. Catanzaro · 2021
Later among the works it cites.
Palm: Scaling language modeling with pathways, 2022
A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, P. Schuh, K. Shi, S. Tsvyashchenko, J. Maynez, A. Rao, P. Barnes, Y. Tay, N. Shazeer, V. Prabhakaran, E. Reif, N. Du, B. Hutchinson, R. Pope, J. Bradbury, J. Austin, M. Isard, G. Gur-Ari, P. Yin, T. Duke, A. Levskaya, S. Ghemawat, S. Dev, H. Michalewski, X. Garcia, V. Misra, K. Robinson, L. Fedus, D. Zhou, D. Ippolito, D. Luan, H. Lim, B. Zoph, A. Spiridonov, R. Sepassi, D. Dohan, S. Agrawal, M. Omernick, A. M. Dai, T. S. Pillai, M. Pellat, A. Lewkowycz, E. Moreira, R. Child, O. Polozov, K. Lee, Z. Zhou, X. Wang, B. Saeta, M. Diaz, O. Firat, M. Catasta, J. Wei, K. Meier-Hellstern, D. Eck, J. Dean, S. Petrov, and N. Fiedel · 2022

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2023

FLuRKA: Fast and accurate unified Low-Rank & Kernel Attention

Gupta, Ahan, Guo, Hao, Yuan, Yueming et al.

Understand

Many efficient $\textit{approximate}$ self-attention techniques have become prevalent since the inception of the transformer architecture.

Two popular classes of these techniques are low-rank and kernel methods.
Each of these methods has its strengths.
We observe these strengths synergistically complement each other and exploit them to fuse low-rank and kernel methods, producing a new class of transformers: FLuRKA ($\textbf{F}$ast $\textbf{L}$ow-$\textbf{R}$ank & $\textbf{K}$ernel$ \textbf{A}$ttention).

Built on

Generating long sequences with sparse transformers
Original
R. Child, S. Gray, A. Radford, and I. Sutskever · 1904
Earlier work this paper cites.
Adaptively sparse transformers
Original
G. M. Correia, V. Niculae, and A. F. T. Martins · 1909
Earlier work this paper cites.
Blockwise self-attention for long document understanding
Original
J. Qiu, H. Ma, O. Levy, S. W. Yih, S. Wang, and J. Tang · 1911
Earlier work this paper cites.
Pytorch: An imperative style, high-performance deep learning library
Original
A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Z. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala · 1912
Earlier work this paper cites.
Reformer: The efficient transformer
Original
N. Kitaev, L. Kaiser, and A. Levskaya · 2001
Earlier work this paper cites.
Sparse sinkhorn attention
Original
Y. Tay, D. Bahri, L. Yang, D. Metzler, and D. Juan · 2002
Earlier work this paper cites.
Efficient content-based sparse attention with routing transformers
Original
A. Roy, M. Saffar, A. Vaswani, and D. Grangier · 2003
Earlier work this paper cites.
Longformer: The long-document transformer
Original
I. Beltagy, M. E. Peters, and A. Cohan · 2004
Earlier work this paper cites.
Language models are few-shot learners
Original
T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei · 2005
Earlier work this paper cites.
Synthesizer: Rethinking self-attention in transformer models
Original
Y. Tay, D. Bahri, D. Metzler, D. Juan, Z. Zhao, and C. Zheng · 2005
Earlier work this paper cites.
Transformers are rnns: Fast autoregressive transformers with linear attention
Original
A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret · 2006
Earlier work this paper cites.
Linformer: Self-attention with linear complexity
Original
S. Wang, B. Z. Li, M. Khabsa, H. Fang, and H. Ma · 2006
Earlier work this paper cites.

Similar

Random features for large-scale kernel machines
A. Rahimi and B. Recht · 2007
Cited alongside, same era.
Big bird: Transformers for longer sequences
Original
M. Zaheer, G. Guruganesh, A. Dubey, J. Ainslie, C. Alberti, S. Ontañón, P. Pham, A. Ravula, Q. Wang, L. Yang, and A. Ahmed · 2007
Cited alongside, same era.
Efficient transformers: A survey
Original
Y. Tay, M. Dehghani, D. Bahri, and D. Metzler · 2009
Cited alongside, same era.
Aligning books and movies: Towards story-like visual explanations by watching movies and reading books
Original
Y. Zhu, R. Kiros, R. Zemel, R. Salakhutdinov, R. Urtasun, A. Torralba, and S. Fidler · 2015
Cited alongside, same era.
Pointer sentinel mixture models
Original
S. Merity, C. Xiong, J. Bradbury, and R. Socher · 2016

Then

Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity
Original
W. Fedus, B. Zoph, and N. Shazeer · 2021
Later among the works it cites.
Random feature attention
H. Peng, N. Pappas, D. Yogatama, R. Schwartz, N. Smith, and L. Kong · 2021
Later among the works it cites.
Nyströmformer: A nyström-based algorithm for approximating self-attention
Original
Y. Xiong, Z. Zeng, R. Chakraborty, M. Tan, G. Fung, Y. Li, and V. Singh · 2021
Later among the works it cites.
Long-short transformer: Efficient transformers for language and vision
Original
C. Zhu, W. Ping, C. Xiao, M. Shoeybi, T. Goldstein, A. Anandkumar, and B. Catanzaro · 2021
Later among the works it cites.
Palm: Scaling language modeling with pathways, 2022
A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, P. Schuh, K. Shi, S. Tsvyashchenko, J. Maynez, A. Rao, P. Barnes, Y. Tay, N. Shazeer, V. Prabhakaran, E. Reif, N. Du, B. Hutchinson, R. Pope, J. Bradbury, J. Austin, M. Isard, G. Gur-Ari, P. Yin, T. Duke, A. Levskaya, S. Ghemawat, S. Dev, H. Michalewski, X. Garcia, V. Misra, K. Robinson, L. Fedus, D. Zhou, D. Ippolito, D. Luan, H. Lim, B. Zoph, A. Spiridonov, R. Sepassi, D. Dohan, S. Agrawal, M. Omernick, A. M. Dai, T. S. Pillai, M. Pellat, A. Lewkowycz, E. Moreira, R. Child, O. Polozov, K. Lee, Z. Zhou, X. Wang, B. Saeta, M. Diaz, O. Firat, M. Catasta, J. Wei, K. Meier-Hellstern, D. Eck, J. Dean, S. Petrov, and N. Fiedel · 2022

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Cited alongside, same era.

Attention is all you need

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin · 2017

Cited alongside, same era.

JAX: composable transformations of Python+NumPy programs, 2018

J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, J. VanderPlas, S. Wanderman-Milne, and Q. Zhang · 2018

Cited alongside, same era.

BERT: pre-training of deep bidirectional transformers for language understanding

Original

J. Devlin, M. Chang, K. Lee, and K. Toutanova · 2018

Cited alongside, same era.

GLUE: A multi-task benchmark and analysis platform for natural language understanding

Original

A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman · 2018

Cited alongside, same era.

Exploring the limits of transfer learning with a unified text-to-text transformer

C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu · 2020

Cited alongside, same era.

Scatterbrain: Unifying sparse and low-rank attention approximation

Original

B. Chen, T. Dao, E. Winsor, Z. Song, A. Rudra, and C. Ré · 2021

Cited alongside, same era.

Rethinking attention with performers

K. M. Choromanski, V. Likhosherstov, D. Dohan, X. Song, A. Gane, T. Sarlos, P. Hawkins, J. Q. Davis, A. Mohiuddin, L. Kaiser, D. B. Belanger, L. J. Colwell, and A. Weller · 2021

Cited alongside, same era.