2020

A$^3$: Accelerating Attention Mechanisms in Neural Networks with Approximation

Ham, Tae Jun, Jung, Sung Jun, Kim, Seonghak et al.

Understand

With the increasing computational demands of neural networks, many hardware accelerators for the neural networks have been proposed.

Such existing neural network accelerators often focus on popular neural network types such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs); however, not much attention has been paid to attention mechanisms, an emerging neural network primitive that enables neural networks to retrieve most relevant information from a knowledge-base, external memory, or past states.
The attention mechanism is widely adopted by many state-of-the-art neural networks for computer vision, natural language processing, and machine translation, and accounts for a large portion of total execution time.
We observe today's practice of implementing this mechanism using matrix-vector multiplication is suboptimal as the attention mechanism is semantically a content-based search where a large portion of computations ends up not being used.

Built on

I. Bello, B. Zoph, A. Vaswani, J. Shlens, and Q. V. Le, “Attention augmented convolutional networks,” CoRR , 2019. [Online]. Available: http://arxiv.org/abs/1904.09925
Original
1904
Earlier work this paper cites.
D. Mohapatra, V. K. Chippa, A. Raghunathan, and K. Roy, “Design of voltage-scalable meta-functions for approximate computing,” in Design, Automation Test in Europe , DATE, March 2011, pp. 1–6
2011
Earlier work this paper cites.
P. Ram and A. G. Gray, “Maximum inner-product search using cone trees,” in ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , KDD, 2012
2012
Earlier work this paper cites.
T. Mikolov, I. Sutskever, K. Chen, G. Corrado, and J. Dean, “Distributed representations of words and phrases and their compositionality,” in International Conference on Neural Information Processing Systems , NIPS, 2013
2013
Earlier work this paper cites.
T. Chen, Z. Du, N. Sun, J. Wang, C. Wu, Y. Chen, and O. Temam, “Diannao: A small-footprint high-throughput accelerator for ubiquitous machine-learning,” in International Conference on Architectural Support for Programming Languages and Operating Systems , ASPLOS, 2014
2014
Earlier work this paper cites.
D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” CoRR , 2014
2014
Earlier work this paper cites.
A. Graves, G. Wayne, and I. Danihelka, “Neural turing machines,” CoRR , 2014
2014
Earlier work this paper cites.
J. Pennington, R. Socher, and C. D. Manning, “Glove: Global vectors for word representation,” in Conference on Empirical Methods in Natural Language Processing , EMNLP, 2014
2014
Earlier work this paper cites.
V. Mnih, N. Heess, A. Graves, and K. Kavukcuoglu, “Recurrent models of visual attention,” in International Conference on Neural Information Processing Systems , NIPS, 2014
2014
Earlier work this paper cites.
A. Shrivastava and P. Li, “Asymmetric LSH (ALSH) for sublinear time maximum inner product search (MIPS),” in International Conference on Neural Information Processing Systems , NIPS, 2014
2014
Earlier work this paper cites.
Y. Chen, T. Luo, S. Liu, S. Zhang, L. He, J. Wang, L. Li, T. Chen, Z. Xu, N. Sun, and O. Temam, “Dadiannao: A machine-learning supercomputer,” in IEEE/ACM International Symposium on Microarchitecture , MICRO, 2014
2014
Earlier work this paper cites.
K. Xu, J. Ba, R. Kiros, K. Cho, A. C. Courville, R. Salakhutdinov, R. S. Zemel, and Y. Bengio, “Show, attend and tell: Neural image caption generation with visual attention,” in International Conference on Machine Learning, ICML , 2015
2015
Earlier work this paper cites.
S. Sukhbaatar, J. Weston, R. Fergus et al. , “End-to-end memory networks,” in International Conference on Neural Information Processing Systems , NIPS, 2015
2015
Earlier work this paper cites.
M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. Corrado, A. Davis, J. Dean, M. Devin et al. , “Tensorflow: Large-scale machine learning on heterogeneous distributed systems,” CoRR , 2015
2015
Earlier work this paper cites.
Facebook, “The bAbI project,” https://research.fb.com/downloads/babi
2015
Earlier work this paper cites.
K. M. Hermann, T. Kociský, E. Grefenstette, L. Espeholt, W. Kay, M. Suleyman, and P. Blunsom, “Teaching machines to read and comprehend,” in Advances in Neural Information Processing Systems , NIPS, 2015
2015
Earlier work this paper cites.
A. M. Rush, S. Chopra, and J. Weston, “A neural attention model for abstractive sentence summarization,” in Conference on Empirical Methods in Natural Language Processing , EMNLP, 2015
2015
Earlier work this paper cites.
M. Jaderberg, K. Simonyan, A. Zisserman, and K. Kavukcuoglu, “Spatial transformer networks,” in International Conference on Neural Information Processing Systems , NIPS, 2015
2015
Earlier work this paper cites.
A. Auvolat and P. Vincent, “Clustering is efficient for approximate maximum inner product search,” CoRR , 2015
2015
Earlier work this paper cites.
Z. Du, R. Fasthuber, T. Chen, P. Ienne, L. Li, T. Luo, X. Feng, Y. Chen, and O. Temam, “Shidiannao: Shifting vision processing closer to the sensor,” in International Symposium on Computer Architecture , ISCA, 2015
2015
Earlier work this paper cites.
D. Liu, T. Chen, S. Liu, J. Zhou, S. Zhou, O. Teman, X. Feng, X. Zhou, and Y. Chen, “Pudiannao: A polyvalent machine learning accelerator,” in International Conference on Architectural Support for Programming Languages and Operating Systems , ASPLOS, 2015
2015
Earlier work this paper cites.
C. Zhang, P. Li, G. Sun, Y. Guan, B. Xiao, and J. Cong, “Optimizing FPGA-based accelerator design for deep convolutional neural networks,” in ACM/SIGDA International Symposium on Field-Programmable Gate Arrays , FPGA, 2015
2015
Earlier work this paper cites.
M. Tapaswi, Y. Zhu, R. Stiefelhagen, A. Torralba, R. Urtasun, and S. Fidler, “MovieQA: Understanding Stories in Movies through Question-Answering,” in Conference on Computer Vision and Pattern Recognition , CVPR, 2016
2016
Earlier work this paper cites.
A. Graves, G. Wayne, M. Reynolds, T. Harley, I. Danihelka, A. Grabska-Barwinska, S. G. Colmenarejo, E. Grefenstette, T. Ramalho, J. Agapiou, A. P. Badia, K. M. Hermann, Y. Zwols, G. Ostrovski, A. Cain, H. King, C. Summerfield, P. Blunsom, K. Kavukcuoglu, and D. Hassabis, “Hybrid computing using a neural network with dynamic external memory,” Nature , 2016
2016
Earlier work this paper cites.
A. P. Parikh, O. Täckström, D. Das, and J. Uszkoreit, “A decomposable attention model for natural language inference,” in Empirical Methods in Natural Language Processing , EMNLP, 2016
2016
Earlier work this paper cites.

Similar

J. Weston, A. Bordes, S. Chopra, A. M. Rush, B. van Merriënboer, A. Joulin, and T. Mikolov, “Towards AI-complete question answering: A set of prerequisite toy tasks,” in International Conference on Learning Representations , ICLR, 2016
2016
Cited alongside, same era.
A. Miller, A. Fisch, J. Dodge, A.-H. Karimi, A. Bordes, and J. Weston, “Key-value memory networks for directly reading documents,” in Empirical Methods in Natural Language Processing , EMNLP, 2016
2016
Cited alongside, same era.
P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang, “SQuAD: 100,000+ questions for machine comprehension of text,” in Empirical Methods in Natural Language Processing , EMNLP, 2016
2016
Cited alongside, same era.
S. Arora, “Word embeddings: Explaining their properties,” https://www.offconvex.org/2016/02/14/word-embeddings-2/
2016
Cited alongside, same era.
B. Wang, K. Liu, and J. Zhao, “Inner attention based recurrent neural networks for answer selection,” in Annual Meeting of the Association for Computational Linguistics , ACL, 2016
2016
Cited alongside, same era.
T. Rocktäschel, E. Grefenstette, K. M. Hermann, T. Koc̆iský, and P. Blunsom, “Reasoning about entailment with neural attention,” in International Conference on Learning Representations , ICLR, 2016
2016
Cited alongside, same era.
S. Chopra, M. Auli, and A. M. Rush, “Abstractive sentence summarization with attentive recurrent neural networks,” in The Conference of the North American Chapter of the Association for Computational Linguistics , NAACL, 2016
2016
Cited alongside, same era.
Z. Yang, D. Yang, C. Dyer, X. He, A. J. Smola, and E. H. Hovy, “Hierarchical attention networks for document classification,” in The Conference of the North American Chapter of the Association for Computational Linguistics , NAACL, 2016
2016
Cited alongside, same era.
Q. You, H. Jin, Z. Wang, C. Fang, and J. Luo, “Image captioning with semantic attention,” in Conference on Computer Vision and Pattern Recognition , CVPR, 2016
2016
Cited alongside, same era.
S. Sharma, R. Kiros, and R. Salakhutdinov, “Action recognition using visual attention,” in International Conference on Learning Representations , ICLR, 2016
2016
Cited alongside, same era.
J. Kuen, Z. Wang, and G. Wang, “Recurrent attentional networks for saliency detection,” in Conference on Computer Vision and Pattern Recognition , CVPR, 2016
2016
Cited alongside, same era.
J. Rae, J. J. Hunt, I. Danihelka, T. Harley, A. W. Senior, G. Wayne, A. Graves, and T. Lillicrap, “Scaling memory-augmented neural networks with sparse reads and writes,” in International Conference on Neural Information Processing Systems , NIPS, 2016
2016
Cited alongside, same era.

Then

V. Khuc, “MemN2N-babi-python,” https://github.com/vinhkhuc/MemN2N-babi-python
2017
Later among the works it cites.
Facebook, “Key-value memory networks for directly reading documents,” https://github.com/facebook/MemNN/tree/master/KVmemnn
2017
Later among the works it cites.
B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le, “Learning transferable architectures for scalable image recognition,” CoRR , 2017. [Online]. Available: http://arxiv.org/abs/1707.07012
Original
2017
Later among the works it cites.
Intel, “Intel Xeon gold 6128 processor,” https://ark.intel.com/products/120482/Intel-Xeon-Gold-6128-Processor-19-25M-Cache-3-40-GHz-
2017
Later among the works it cites.
M. J. Seo, A. Kembhavi, A. Farhadi, and H. Hajishirzi, “Bidirectional attention flow for machine comprehension,” in International Conference on Learning Representations , ICLR, 2017
2017
Later among the works it cites.
N. Pappas and A. Popescu-Belis, “Multilingual hierarchical attention networks for document classification,” in International Joint Conference on Natural Language Processing , IJCNLP, 2017
2017
Later among the works it cites.
K. Kim, M. Heo, S. Choi, and B. Zhang, “Deepstory: Video story QA by deep embedded memory networks,” in International Joint Conference on Artificial Intelligence , IJCAI, 2017
2017
Later among the works it cites.
F. Wang, M. Jiang, C. Qian, S. Yang, C. Li, H. Zhang, X. Wang, and X. Tang, “Residual attention network for image classification,” in Conference on Computer Vision and Pattern Recognition , CVPR, 2017
2017
Later among the works it cites.
S. Han, J. Kang, H. Mao, Y. Hu, X. Li, Y. Li, D. Xie, H. Luo, S. Yao, Y. Wang, H. Yang, and W. B. J. Dally, “ESE: Efficient speech recognition engine with sparse LSTM on FPGA,” in ACM/SIGDA International Symposium on Field-Programmable Gate Arrays , FPGA, 2017
2017
Later among the works it cites.
A. Parashar, M. Rhu, A. Mukkara, A. Puglielli, R. Venkatesan, B. Khailany, J. Emer, S. W. Keckler, and W. J. Dally, “SCNN: An accelerator for compressed-sparse convolutional neural networks,” in International Symposium on Computer Architecture , ISCA, 2017
2017
Later among the works it cites.
A. Raha, S. Venkataramani, V. Raghunathan, and A. Raghunathan, “Energy-efficient reduce-and-rank using input-adaptive approximations,” IEEE Transactions on Very Large Scale Integration Systems , vol. 25, no. 2, pp. 462–475, Feb 2017
2017
Later among the works it cites.
A. Raha and V. Raghunathan, “Qlut: Input-aware quantized table lookup for energy-efficient approximate accelerators,” ACM Transactions on Embedded Computing Systems , vol. 16, no. 5s, 2017
2017
Later among the works it cites.
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” CoRR , 2018
2018
Later among the works it cites.
Z. Yin and Y. Shen, “On the dimensionality of word embedding,” in International Conference on Neural Information Processing Systems , NIPS, 2018
2018
Later among the works it cites.
G. Research, “Tensorflow code and pre-trained models for BERT.” https://github.com/google-research/bert
2018
Later among the works it cites.
NVIDIA, “NVIDIA TITAN V,” https://www.nvidia.com/en-us/titan/titan-v/
2018
Later among the works it cites.
NVIDIA, “Inside Volta: The world’s most advanced data center GPU,” https://devblogs.nvidia.com/inside-volta/
2018
Later among the works it cites.
J. Fowers, K. Ovtcharov, M. Papamichael, T. Massengill, M. Liu, D. Lo, S. Alkalay, M. Haselman, L. Adams, M. Ghandi, S. Heil, P. Patel, A. Sapek, G. Weisz, L. Woods, S. Lanka, S. K. Reinhardt, A. M. Caulfield, E. S. Chung, and D. Burger, “A configurable cloud-scale DNN processor for real-time AI,” in International Symposium on Computer Architecture , ISCA, 2018
2018
Later among the works it cites.
J. Park, J. Kung, W. Yi, and J. Kim, “Maximizing system performance by balancing computation loads in LSTM accelerators,” in Design, Automation Test in Europe , DATE, 2018
2018
Later among the works it cites.
C. Gao, D. Neil, E. Ceolini, S.-C. Liu, and T. Delbruck, “DeltaRNN: A power-efficient Recurrent Neural Network accelerator,” in ACM/SIGDA International Symposium on Field-Programmable Gate Arrays , FPGA, 2018
2018
Later among the works it cites.
V. Akhlaghi, A. Yazdanbakhsh, K. Samadi, R. K. Gupta, and H. Esmaeilzadeh, “Snapea: Predictive early activation for reducing computation in deep convolutional neural networks,” in International Symposium on Computer Architecture , ISCA, 2018
2018
Later among the works it cites.
E. Real, A. Aggarwal, Y. Huang, and Q. V. Le, “Regularized evolution for image classifier architecture search,” in AAAI Conference on Artificial Intelligence , AAAI, 2019
2019
Later among the works it cites.
Wikichip, “Skylake (server) - microarchitectures - Intel,” https://en.wikichip.org/wiki/intel/microarchitectures/skylake_(server)
2019
Later among the works it cites.
M. Masadeh, O. Hasan, and S. Tahar, “Input-conscious approximate multiply-accumulate (mac) unit for energy-efficiency,” IEEE Access , vol. 7, 2019
2019
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…