2018

Interstellar: Using Halide's Scheduling Language to Analyze DNN Accelerators

Yang, Xuan, Gao, Mingyu, Liu, Qiaoyi et al.

Understand

We show that DNN accelerator micro-architectures and their program mappings represent specific choices of loop order and hardware parallelism for computing the seven nested loops of DNNs, which enables us to create a formal taxonomy of all existing dense DNN accelerators.

Surprisingly, the loop transformations needed to create these hardware variants can be precisely and concisely represented by Halide's scheduling language.
By modifying the Halide compiler to generate hardware, we create a system that can fairly compare these prior accelerators.
As long as proper loop blocking schemes are used, and the hardware can support mapping replicated loops, many different hardware dataflows yield similar energy efficiency with good performance.

Built on

A fast learning algorithm for deep belief nets
Geoffrey E Hinton, Simon Osindero, and Yee-Whye Teh · 2006
Earlier work this paper cites.
Optimizing NUCA organizations and wiring alternatives for large caches with CACTI 6.0
Naveen Muralimanohar, Rajeev Balasubramonian, and Norm Jouppi · 2007
Earlier work this paper cites.
Neuflow: A runtime reconfigurable dataflow processor for vision
Clément Farabet, Berin Martini, Benoit Corda, Polina Akselrod, Eugenio Culurciello, and Yann LeCun · 2011
Earlier work this paper cites.
Halide: A language and compiler for optimizing parallelism, locality, and recomputation in image processing pipelines
Jonathan Ragan-Kelley, Connelly Barnes, Andrew Adams, Sylvain Paris, Frédo Durand, and Saman Amarasinghe · 2013
Earlier work this paper cites.
Improving high level synthesis optimization opportunity through polyhedral transformations
Wei Zuo, Yun Liang, Peng Li, Kyle Rupnow, Deming Chen, and Jason Cong · 2013
Earlier work this paper cites.
DianNao: A small-footprint high-throughput accelerator for ubiquitous machine-learning
Tianshi Chen, Zidong Du, Ninghui Sun, Jia Wang, Chengyong Wu, Yunji Chen, and Olivier Temam · 2014
Earlier work this paper cites.
DaDianNao: A machine-learning supercomputer
Yunji Chen, Tao Luo, Shaoli Liu, Shijin Zhang, Liqiang He, Jia Wang, Ling Li, Tianshi Chen, Zhiwei Xu, Ninghui Sun, and Olivier Temam · 2014
Earlier work this paper cites.
A 240 G-ops/s mobile coprocessor for deep neural networks
Vinayak Gokhale, Jonghoon Jin, Aysegul Dundar, Berin Martini, and Eugenio Culurciello · 2014
Earlier work this paper cites.
Very deep convolutional networks for large-scale image recognition
Original
Karen Simonyan and Andrew Zisserman · 2014
Earlier work this paper cites.
Sequence to sequence learning with neural networks
Original
Ilya Sutskever, Oriol Vinyals, and Quoc V. Le · 2014
Earlier work this paper cites.
Going deeper with convolutions
Original
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich · 2014
Earlier work this paper cites.
ShiDianNao: Shifting vision processing closer to the sensor
Zidong Du, Robert Fasthuber, Tianshi Chen, Paolo Ienne, Ling Li, Tao Luo, Xiaobing Feng, Yunji Chen, and Olivier Temam · 2015
Earlier work this paper cites.
Polymage: Automatic optimization for image processing pipelines
Ravi Teja Mullapudi, Vinay Vasista, and Uday Bondhugula · 2015
Earlier work this paper cites.
Optimizing FPGA-based accelerator design for deep convolutional neural networks
Chen Zhang, Peng Li, Guangyu Sun, Yijin Guan, Bingjun Xiao, and Jason Cong · 2015
Earlier work this paper cites.
Cnvlutin: Ineffectual-neuron-free deep neural network computing
Jorge Albericio, Patrick Judd, Tayler Hetherington, Tor Aamodt, Natalie Enright Jerger, and Andreas Moshovos · 2016
Earlier work this paper cites.
Fused-layer CNN accelerators
Manoj Alwani, Han Chen, Michael Ferdman, and Peter Milder · 2016
Earlier work this paper cites.

Similar

Eyeriss: A spatial architecture for energy-efficient dataflow for convolutional neural networks
Yu-Hsin Chen, Joel Emer, and Vivienne Sze · 2016
Cited alongside, same era.
Eyeriss: An energy-efficient reconfigurable accelerator for deep convolutional neural networks
Yu-Hsin Chen, Tushar Krishna, Joel Emer, and Vivienne Sze · 2016
Cited alongside, same era.
PRIME: A novel processing-in-memory architecture for neural network computation in ReRAM-based main memory
Ping Chi, Shuangchen Li, Cong Xu, Tao Zhang, Jishen Zhao, Yongpan Liu, Yu Wang, and Yuan Xie · 2016
Cited alongside, same era.
EIE: Efficient inference engine on compressed deep neural network
Song Han, Xingyu Liu, Huizi Mao, Jing Pu, Ardavan Pedram, Mark A. Horowitz, and William J. Dally · 2016
Cited alongside, same era.
Neurocube: A programmable digital neuromorphic architecture with high-density 3D memory
Duckhwan Kim, Jaeha Kung, Sek Chai, Sudhakar Yalamanchili, and Saibal Mukhopadhyay · 2016
Cited alongside, same era.
A high performance FPGA-based accelerator for large-scale convolutional neural networks
Huimin Li, Xitian Fan, Li Jiao, Wei Cao, Xuegong Zhou, and Lingli Wang · 2016
Cited alongside, same era.
Going deeper with embedded FPGA platform for convolutional neural network
Jiantao Qiu, Jie Wang, Song Yao, Kaiyuan Guo, Boxun Li, Erjin Zhou, Jincheng Yu, Tianqi Tang, Ningyi Xu, Sen Song, et al · 2016
Cited alongside, same era.
From high-level deep neural models to FPGAs
H. Sharma, J. Park, D. Mahajan, E. Amaro, J. K. Kim, C. Shao, A. Mishra, and H. Esmaeilzadeh · 2016
Cited alongside, same era.
Overcoming resource underutilization in spatial CNN accelerators
Yongming Shen, Mechael Ferdman, and Peter Milder · 2016
Cited alongside, same era.
Throughput-optimized OpenCL-based FPGA accelerator for large-scale convolutional neural networks
Naveen Suda, Vikas Chandra, Ganesh Dasika, Abinash Mohanty, Yufei Ma, Sarma Vrudhula, Jae-sun Seo, and Yu Cao · 2016
Cited alongside, same era.
A systematic approach to blocking convolutional neural networks
Original
Xuan Yang, Jing Pu, Blaine Burton Rister, Nikhil Bhagdikar, Stephen Richardson, Shahar Kvatinsky, Jonathan Ragan-Kelley, Ardavan Pedram, and Mark Horowitz · 2016
Cited alongside, same era.
Cambricon-X: An accelerator for sparse neural networks
Shijin Zhang, Zidong Du, Lei Zhang, Huiying Lan, Shaoli Liu, Ling Li, Qi Guo, Tianshi Chen, and Yunji Chen · 2016
Cited alongside, same era.

Then

FlexFlow: A flexible dataflow accelerator architecture for convolutional neural networks
Wenyan Lu, Guihai Yan, Jiajun Li, Shijun Gong, Yinhe Han, and Xiaowei Li · 2017
Later among the works it cites.
SCNN: An accelerator for compressed-sparse convolutional neural networks
Angshuman Parashar, Minsoo Rhu, Anurag Mukkara, Antonio Puglielli, Rangharajan Venkatesan, Brucek Khailany, Joel Emer, Stephen W Keckler, and William J Dally · 2017
Later among the works it cites.
Programming heterogeneous systems from an image processing DSL
Jing Pu, Steven Bell, Xuan Yang, Jeff Setter, Stephen Richardson, Jonathan Ragan-Kelley, and Mark Horowitz · 2017
Later among the works it cites.
Maximizing CNN accelerator efficiency through resource partitioning
Yongming Shen, Mechael Ferdman, and Peter Milder · 2017
Later among the works it cites.
Automated systolic array architecture synthesis for high throughput CNN inference on FPGAs
Xuechao Wei, Cody Hao Yu, Peng Zhang, Youxiang Chen, Yuxin Wang, Han Hu, Yun Liang, and Jason Cong · 2017
Later among the works it cites.
Scalpel: Customizing DNN pruning to the underlying hardware parallelism
Jiecao Yu, Andrew Lukefahr, David Palframan, Ganesh Dasika, Reetuparna Das, and Scott Mahlke · 2017
Later among the works it cites.
Frequency domain acceleration of convolutional neural networks on CPU-FPGA shared memory system
Chi Zhang and Viktor Prasanna · 2017
Later among the works it cites.
MAESTRO: an open-source infrastructure for modeling dataflows within deep learning accelerators
Original
Hyoukjun Kwon, Michael Pellauer, and Tushar Krishna · 2018
Closest in time.
MAERI: Enabling flexible dataflow mapping over DNN accelerators via reconfigurable interconnects
Hyoukjun Kwon, Ananda Samajdar, and Tushar Krishna · 2018
Closest in time.
Towards efficient microarchitectural design for accelerating unsupervised GAN-based deep learning
Mingcong Song, Jiaqi Zhang, Huixiang Chen, and Tao Li · 2018
Closest in time.
Eyeriss v2: A flexible accelerator for emerging deep neural networks on mobile devices
Y. Chen, T. Yang, J. Emer, and V. Sze · 2019
Closest in time.
Tangram: Optimized coarse-grained dataflow for scalable nn accelerators
Mingyu Gao, Xuan Yang, Jing Pu, Mark Horowitz, and Christos Kozyrakis · 2019
Closest in time.
Understanding reuse, performance, and hardware cost of dnn dataflow: A data-centric approach
Hyoukjun Kwon, Prasanth Chatarasi, Michael Pellauer, Angshuman Parashar, Vivek Sarkar, and Tushar Krishna · 2019
Closest in time.
Timeloop: A systematic approach to dnn accelerator evaluation
A. Parashar, P. Raina, Y. S. Shao, Y. Chen, V. A. Ying, A. Mukkara, R. Venkatesan, B. Khailany, S. W. Keckler, and J. Emer · 2019
Closest in time.
A Systematic Framework to Analyze the Design Space of DNN Accelerators
Xuan Yang · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…