Built on

A. Pinar and M. T. Heath, “Improving performance of sparse matrix-vector multiplication,” in SC’99: Proceedings of the 1999 ACM/IEEE Conference on Supercomputing . IEEE, 1999, pp. 30–30
1999
Earlier work this paper cites.
R. Vuduc, J. W. Demmel, and K. A. Yelick, “Oski: A library of automatically tuned sparse matrix kernels,” in Journal of Physics: Conference Series , vol. 16, no. 1. IOP Publishing, 2005, p. 071
2005
Earlier work this paper cites.
D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” arXiv preprint arXiv:1409.0473 , 2014
Original
2014
Earlier work this paper cites.
S. Han, H. Mao, and W. J. Dally, “Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding,” arXiv preprint arXiv:1510.00149 , 2015
Original
2015
Earlier work this paper cites.
M. Rastegari, V. Ordonez, J. Redmon, and A. Farhadi, “Xnor-net: Imagenet classification using binary convolutional neural networks,” in European conference on computer vision . Springer, 2016, pp. 525–542
2016
Earlier work this paper cites.
S. Han, X. Liu, H. Mao, J. Pu, A. Pedram, M. A. Horowitz, and W. J. Dally, “Eie: Efficient inference engine on compressed deep neural network,” ACM SIGARCH Computer Architecture News , vol. 44, no. 3, pp. 243–254, 2016
2016
Earlier work this paper cites.
J. Hestness, S. Narang, N. Ardalani, G. Diamos, H. Jun, H. Kianinejad, M. Patwary, M. Ali, Y. Yang, and Y. Zhou, “Deep learning scaling is predictable, empirically,” arXiv preprint arXiv:1712.00409 , 2017
Original
2017
Earlier work this paper cites.
NVIDIA, “Nvidia tesla v100 gpu architecture,” 2017
2017
Earlier work this paper cites.
P. Micikevicius, S. Narang, J. Alben, G. Diamos, E. Elsen, D. Garcia, B. Ginsburg, M. Houston, O. Kuchaiev, G. Venkatesh et al. , “Mixed precision training,” arXiv preprint arXiv:1710.03740 , 2017
Original
2017
Earlier work this paper cites.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems , vol. 30, 2017
2017
Earlier work this paper cites.
H. Mao, S. Han, J. Pool, W. Li, X. Liu, Y. Wang, and W. J. Dally, “Exploring the regularity of sparse structure in convolutional neural networks,” arXiv preprint arXiv:1705.08922 , 2017
Original
2017
Earlier work this paper cites.
OpenAI, “AI and compute,” 2018. [Online]. Available: https://openai.com/blog/ai-and-compute/
2018
Earlier work this paper cites.
Z. Jia, M. Maggioni, B. Staiger, and D. P. Scarpazza, “Dissecting the nvidia volta gpu architecture via microbenchmarking,” arXiv preprint arXiv:1804.06826 , 2018
Original
2018
Earlier work this paper cites.
F. Tung and G. Mori, “Deep neural network compression by in-parallel pruning-quantization,” IEEE transactions on pattern analysis and machine intelligence , vol. 42, no. 3, pp. 568–579, 2018
2018
Earlier work this paper cites.
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805 , 2018
Original
2018
Earlier work this paper cites.
X. Wang, W. Liu, W. Xue, and L. Wu, “swsptrsv: a fast sparse triangular solve with sparse level tile layout on sunway architectures,” in Proceedings of the 23rd ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming , 2018, pp. 338–353
2018
Earlier work this paper cites.
Y. Chen, K. Li, W. Yang, G. Xiao, X. Xie, and T. Li, “Performance-aware model for sparse matrix-matrix multiplication on the sunway taihulight supercomputer,” IEEE transactions on parallel and distributed systems , vol. 30, no. 4, pp. 923–938, 2018
2018
Earlier work this paper cites.
D. Zhang, J. Yang, D. Ye, and G. Hua, “Lq-nets: Learned quantization for highly accurate and compact deep neural networks,” in Proceedings of the European conference on computer vision (ECCV) , 2018, pp. 365–382
2018
Earlier work this paper cites.
J. S. Rosenfeld, A. Rosenfeld, Y. Belinkov, and N. Shavit, “A constructive prediction of the generalization error across scales,” arXiv preprint arXiv:1909.12673 , 2019
Original
2019
Earlier work this paper cites.
E. Strubell, A. Ganesh, and A. McCallum, “Energy and policy considerations for deep learning in NLP,” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL) , 2019
2019
Earlier work this paper cites.
R. Child, S. Gray, A. Radford, and I. Sutskever, “Generating long sequences with sparse transformers,” arXiv preprint arXiv:1904.10509 , 2019
Original
2019
Earlier work this paper cites.
G. Srivastava, D. Kadetotad, S. Yin, V. Berisha, C. Chakrabarti, and J.-s. Seo, “Joint optimization of quantization and structured sparsity for compressed deep neural networks,” in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2019, pp. 1393–1397
2019
Earlier work this paper cites.
H. You, C. Li, P. Xu, Y. Fu, Y. Wang, X. Chen, R. G. Baraniuk, Z. Wang, and Y. Lin, “Drawing early-bird tickets: Towards more efficient training of deep networks,” arXiv preprint arXiv:1909.11957 , 2019
Original
2019
Earlier work this paper cites.
A. Bhandare, V. Sripathi, D. Karkada, V. Menon, S. Choi, K. Datta, and V. Saletore, “Efficient 8-bit quantization of transformer neural machine language translation model,” arXiv preprint arXiv:1906.00532 , 2019
Original
2019
Earlier work this paper cites.
O. Zafrir, G. Boudoukh, P. Izsak, and M. Wasserblat, “Q8bert: Quantized 8bit bert,” in 2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing-NeurIPS Edition (EMC2-NIPS) . IEEE, 2019, pp. 36–39
2019
Earlier work this paper cites.
A. Li, T. Geng, T. Wang, M. Herbordt, S. L. Song, and K. Barker, “Bstc: A novel binarized-soft-tensor-core design for accelerating bit-based approximated neural nets,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis , 2019, pp. 1–30
2019
Earlier work this paper cites.

Similar

Z. Xie, G. Tan, W. Liu, and N. Sun, “Ia-spgemm: An input-aware auto-tuning framework for parallel sparse matrix-matrix multiplication,” in Proceedings of the ACM International Conference on Supercomputing , 2019, pp. 94–105
2019
Cited alongside, same era.
K. Wang, Z. Liu, Y. Lin, J. Lin, and S. Han, “Haq: Hardware-aware automated quantization with mixed precision,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2019, pp. 8612–8620
2019
Cited alongside, same era.
M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro, “Megatron-lm: Training multi-billion parameter language models using model parallelism,” arXiv preprint arXiv:1909.08053 , 2019
Original
2019
Cited alongside, same era.
K. Osawa, Y. Tsuji, Y. Ueno, A. Naruse, R. Yokota, and S. Matsuoka, “Large-scale distributed second-order optimization using kronecker-factored approximate curvature for deep convolutional neural networks,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2019, pp. 12 359–12 367

Then

Z. Chen, Z. Qu, L. Liu, Y. Ding, and Y. Xie, “Efficient tensor core-based gpu kernels for structured sparsity under reduced precision,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis , 2021, pp. 1–14
2021
Later among the works it cites.
NVIDIA, “Tuning CUDA Applications for Ampere,” September 2021, https://docs.nvidia.com/cuda/ampere-tuning-guide/index.html
2021
Later among the works it cites.
I. Hubara, Y. Nahshan, Y. Hanani, R. Banner, and D. Soudry, “Accurate post training quantization with small calibration sets,” in International Conference on Machine Learning . PMLR, 2021, pp. 4466–4475
2021
Later among the works it cites.
M. Nagel, M. Fournarakis, R. A. Amjad, Y. Bondarenko, M. van Baalen, and T. Blankevoort, “A white paper on neural network quantization,” arXiv preprint arXiv:2106.08295 , 2021
Original
2021
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2022

Efficient Quantized Sparse Matrix Operations on Tensor Cores

Li, Shigang, Osawa, Kazuki, Hoefler, Torsten

Understand

The exponentially growing model size drives the continued success of deep learning, but it brings prohibitive computation and memory cost.

From the algorithm perspective, model sparsification and quantization have been studied to alleviate the problem.
From the architecture perspective, hardware vendors provide Tensor cores for acceleration.
However, it is very challenging to gain practical speedups from sparse, low-precision matrix operations on Tensor cores, because of the strict requirements for data layout and lack of support for efficiently manipulating the low-precision integers.

Built on

A. Pinar and M. T. Heath, “Improving performance of sparse matrix-vector multiplication,” in SC’99: Proceedings of the 1999 ACM/IEEE Conference on Supercomputing . IEEE, 1999, pp. 30–30
1999
Earlier work this paper cites.
R. Vuduc, J. W. Demmel, and K. A. Yelick, “Oski: A library of automatically tuned sparse matrix kernels,” in Journal of Physics: Conference Series , vol. 16, no. 1. IOP Publishing, 2005, p. 071
2005
Earlier work this paper cites.
D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” arXiv preprint arXiv:1409.0473 , 2014
Original
2014
Earlier work this paper cites.
S. Han, H. Mao, and W. J. Dally, “Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding,” arXiv preprint arXiv:1510.00149 , 2015
Original
2015
Earlier work this paper cites.
M. Rastegari, V. Ordonez, J. Redmon, and A. Farhadi, “Xnor-net: Imagenet classification using binary convolutional neural networks,” in European conference on computer vision . Springer, 2016, pp. 525–542
2016
Earlier work this paper cites.
S. Han, X. Liu, H. Mao, J. Pu, A. Pedram, M. A. Horowitz, and W. J. Dally, “Eie: Efficient inference engine on compressed deep neural network,” ACM SIGARCH Computer Architecture News , vol. 44, no. 3, pp. 243–254, 2016
2016
Earlier work this paper cites.
J. Hestness, S. Narang, N. Ardalani, G. Diamos, H. Jun, H. Kianinejad, M. Patwary, M. Ali, Y. Yang, and Y. Zhou, “Deep learning scaling is predictable, empirically,” arXiv preprint arXiv:1712.00409 , 2017
Original
2017
Earlier work this paper cites.
NVIDIA, “Nvidia tesla v100 gpu architecture,” 2017
2017
Earlier work this paper cites.
P. Micikevicius, S. Narang, J. Alben, G. Diamos, E. Elsen, D. Garcia, B. Ginsburg, M. Houston, O. Kuchaiev, G. Venkatesh et al. , “Mixed precision training,” arXiv preprint arXiv:1710.03740 , 2017
Original
2017
Earlier work this paper cites.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems , vol. 30, 2017
2017
Earlier work this paper cites.
H. Mao, S. Han, J. Pool, W. Li, X. Liu, Y. Wang, and W. J. Dally, “Exploring the regularity of sparse structure in convolutional neural networks,” arXiv preprint arXiv:1705.08922 , 2017
Original
2017
Earlier work this paper cites.
OpenAI, “AI and compute,” 2018. [Online]. Available: https://openai.com/blog/ai-and-compute/
2018
Earlier work this paper cites.
Z. Jia, M. Maggioni, B. Staiger, and D. P. Scarpazza, “Dissecting the nvidia volta gpu architecture via microbenchmarking,” arXiv preprint arXiv:1804.06826 , 2018
Original
2018
Earlier work this paper cites.
F. Tung and G. Mori, “Deep neural network compression by in-parallel pruning-quantization,” IEEE transactions on pattern analysis and machine intelligence , vol. 42, no. 3, pp. 568–579, 2018
2018
Earlier work this paper cites.
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805 , 2018
Original
2018
Earlier work this paper cites.
X. Wang, W. Liu, W. Xue, and L. Wu, “swsptrsv: a fast sparse triangular solve with sparse level tile layout on sunway architectures,” in Proceedings of the 23rd ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming , 2018, pp. 338–353
2018
Earlier work this paper cites.
Y. Chen, K. Li, W. Yang, G. Xiao, X. Xie, and T. Li, “Performance-aware model for sparse matrix-matrix multiplication on the sunway taihulight supercomputer,” IEEE transactions on parallel and distributed systems , vol. 30, no. 4, pp. 923–938, 2018
2018
Earlier work this paper cites.
D. Zhang, J. Yang, D. Ye, and G. Hua, “Lq-nets: Learned quantization for highly accurate and compact deep neural networks,” in Proceedings of the European conference on computer vision (ECCV) , 2018, pp. 365–382
2018
Earlier work this paper cites.
J. S. Rosenfeld, A. Rosenfeld, Y. Belinkov, and N. Shavit, “A constructive prediction of the generalization error across scales,” arXiv preprint arXiv:1909.12673 , 2019
Original
2019
Earlier work this paper cites.
E. Strubell, A. Ganesh, and A. McCallum, “Energy and policy considerations for deep learning in NLP,” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL) , 2019
2019
Earlier work this paper cites.
R. Child, S. Gray, A. Radford, and I. Sutskever, “Generating long sequences with sparse transformers,” arXiv preprint arXiv:1904.10509 , 2019
Original
2019
Earlier work this paper cites.
G. Srivastava, D. Kadetotad, S. Yin, V. Berisha, C. Chakrabarti, and J.-s. Seo, “Joint optimization of quantization and structured sparsity for compressed deep neural networks,” in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2019, pp. 1393–1397
2019
Earlier work this paper cites.
H. You, C. Li, P. Xu, Y. Fu, Y. Wang, X. Chen, R. G. Baraniuk, Z. Wang, and Y. Lin, “Drawing early-bird tickets: Towards more efficient training of deep networks,” arXiv preprint arXiv:1909.11957 , 2019
Original
2019
Earlier work this paper cites.
A. Bhandare, V. Sripathi, D. Karkada, V. Menon, S. Choi, K. Datta, and V. Saletore, “Efficient 8-bit quantization of transformer neural machine language translation model,” arXiv preprint arXiv:1906.00532 , 2019
Original
2019
Earlier work this paper cites.
O. Zafrir, G. Boudoukh, P. Izsak, and M. Wasserblat, “Q8bert: Quantized 8bit bert,” in 2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing-NeurIPS Edition (EMC2-NIPS) . IEEE, 2019, pp. 36–39
2019
Earlier work this paper cites.
A. Li, T. Geng, T. Wang, M. Herbordt, S. L. Song, and K. Barker, “Bstc: A novel binarized-soft-tensor-core design for accelerating bit-based approximated neural nets,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis , 2019, pp. 1–30
2019
Earlier work this paper cites.

Similar

Z. Xie, G. Tan, W. Liu, and N. Sun, “Ia-spgemm: An input-aware auto-tuning framework for parallel sparse matrix-matrix multiplication,” in Proceedings of the ACM International Conference on Supercomputing , 2019, pp. 94–105
2019
Cited alongside, same era.
K. Wang, Z. Liu, Y. Lin, J. Lin, and S. Han, “Haq: Hardware-aware automated quantization with mixed precision,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2019, pp. 8612–8620
2019
Cited alongside, same era.
M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro, “Megatron-lm: Training multi-billion parameter language models using model parallelism,” arXiv preprint arXiv:1909.08053 , 2019
Original
2019
Cited alongside, same era.
K. Osawa, Y. Tsuji, Y. Ueno, A. Naruse, R. Yokota, and S. Matsuoka, “Large-scale distributed second-order optimization using kronecker-factored approximate curvature for deep convolutional neural networks,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2019, pp. 12 359–12 367

Then

Z. Chen, Z. Qu, L. Liu, Y. Ding, and Y. Xie, “Efficient tensor core-based gpu kernels for structured sparsity under reduced precision,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis , 2021, pp. 1–14
2021
Later among the works it cites.
NVIDIA, “Tuning CUDA Applications for Ampere,” September 2021, https://docs.nvidia.com/cuda/ampere-tuning-guide/index.html
2021
Later among the works it cites.
I. Hubara, Y. Nahshan, Y. Hanani, R. Banner, and D. Soudry, “Accurate post training quantization with small calibration sets,” in International Conference on Machine Learning . PMLR, 2021, pp. 4466–4475
2021
Later among the works it cites.
M. Nagel, M. Fournarakis, R. A. Amjad, Y. Bondarenko, M. van Baalen, and T. Blankevoort, “A white paper on neural network quantization,” arXiv preprint arXiv:2106.08295 , 2021
Original
2021
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

2019

Cited alongside, same era.

J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, “Scaling laws for neural language models,” arXiv preprint arXiv:2001.08361 , 2020

Original

2020

Cited alongside, same era.

T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al. , “Language models are few-shot learners,” Advances in neural information processing systems , vol. 33, pp. 1877–1901, 2020

2020

Cited alongside, same era.

——, “Exploiting NVIDIA Ampere Structured Sparsity with cuSPARSELt ,” December 2020, https://developer.nvidia.com/blog/exploiting-ampere-structured-sparsity-with-cusparselt/

2020

Cited alongside, same era.

T. Gale, M. Zaharia, C. Young, and E. Elsen, “Sparse gpu kernels for deep learning,” in SC20: International Conference for High Performance Computing, Networking, Storage and Analysis . IEEE, 2020, pp. 1–14

2020

Cited alongside, same era.

M. Van Baalen, C. Louizos, M. Nagel, R. A. Amjad, Y. Wang, T. Blankevoort, and M. Welling, “Bayesian bits: Unifying quantization and pruning,” Advances in neural information processing systems , vol. 33, pp. 5741–5752, 2020

2020

Cited alongside, same era.

H. Yang, S. Gui, Y. Zhu, and J. Liu, “Automatic neural network compression by sparsity-quantization joint learning: A constrained optimization-based approach,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2020, pp. 2178–2188

2020

Cited alongside, same era.

P.-H. Yu, S.-S. Wu, J. P. Klopp, L.-G. Chen, and S.-Y. Chien, “Joint pruning & quantization for extremely sparse neural networks,” arXiv preprint arXiv:2010.01892 , 2020

Original

2020

Cited alongside, same era.

H. Wu, P. Judd, X. Zhang, M. Isaev, and P. Micikevicius, “Integer quantization for deep learning inference: Principles and empirical evaluation,” arXiv preprint arXiv:2004.09602 , 2020

Original

2020

Cited alongside, same era.

F. Lagunas, E. Charlaix, V. Sanh, and A. M. Rush, “Block pruning for faster transformers,” arXiv preprint arXiv:2109.04838 , 2021

Original

2021