HyPar: Towards Hybrid Parallelism for Deep Learning Accelerator Array

Understand

With the rise of artificial intelligence in recent years, Deep Neural Networks (DNNs) have been widely used in many domains.

• To achieve high performance and energy efficiency, hardware acceleration (especially inference) of DNNs is intensively studied both in academia and industry.
• However, we still face two challenges: large DNN models and datasets, which incur frequent off-chip memory accesses; and the training of DNNs, which is not well-explored in recent accelerator designs.
• To truly provide high throughput and energy efficient acceleration for the training of deep and large models, we inevitably need to use multiple accelerators to explore the coarse-grain parallelism, compared to the fine-grain parallelism inside a layer considered in most of the existing architectures.