Striped Attention: Faster Ring Attention for Causal Transformers

Generating long sequences with sparse transformers

Original

Child, R., Gray, S., Radford, A., and Sutskever, I · 1904

Earlier work this paper cites.

Megatron-lm: Training multi-billion parameter language models using model parallelism

Original

Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B · 1909

Earlier work this paper cites.

Linformer: Self-attention with linear complexity

Original

Wang, S., Li, B. Z., Khabsa, M., Fang, H., and Ma, H · 2006

Earlier work this paper cites.

Large scale distributed deep networks

Dean, J., Corrado, G., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M. a., Senior, A., Tucker, P., Yang, K., Le, Q., and Ng, A · 2012

Earlier work this paper cites.

Imagenet classification with deep convolutional neural networks

Krizhevsky, A., Sutskever, I., and Hinton, G. E · 2012

Earlier work this paper cites.

Attention is all you need

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I · 2017

Earlier work this paper cites.

JAX: composable transformations of Python+NumPy programs, 2018

Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., VanderPlas, J., Wanderman-Milne, S., and Zhang, Q · 2018

Earlier work this paper cites.

Gpipe: Efficient training of giant neural networks using pipeline parallelism

Original

Huang, Y., Cheng, Y., Chen, D., Lee, H., Ngiam, J., Le, Q. V., and Chen, Z · 2018

Cited alongside, same era.

Improving language understanding by generative pre-training

Radford, A., Narasimhan, K., Salimans, T., Sutskever, I., et al · 2018

Cited alongside, same era.

Language models are unsupervised multitask learners

Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I · 2019

Cited alongside, same era.

Language models are few-shot learners

Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D · 2020

Cited alongside, same era.

Fully sharded data parallel: Faster ai training with fewer gpus, Jul 2021

Ott, M., Shleifer, S., Xu, M., Goyal, P., Duval, Q., and Caggiano, V · 2021

Flashattention: Fast and memory-efficient exact attention with io-awareness

Dao, T., Fu, D., Ermon, S., Rudra, A., and Ré, C · 2022

Later among the works it cites.

Reducing activation recomputation in large transformer models, 2022

Korthikanti, V., Casper, J., Lym, S., McAfee, L., Andersch, M., Shoeybi, M., and Catanzaro, B · 2022

Later among the works it cites.

Sequence parallelism: Long sequence training from system perspective, 2022

Li, S., Xue, F., Baranwal, C., Li, Y., and You, Y · 2022

Later among the works it cites.

Flashattention-2: Faster attention with better parallelism and work partitioning

Original

Dao, T · 2023

Closest in time.

Deepspeed ulysses: System optimizations for enabling training of extreme long sequence transformer models, 2023

Jacobs, S. A., Tanaka, M., Zhang, C., Zhang, M., Song, S. L., Rajbhandari, S., and He, Y · 2023

Closest in time.