HG-PIPE: Vision Transformer Acceleration with Hybrid-Grained Pipeline

Understand

Vision Transformer (ViT) acceleration with field programmable gate array (FPGA) is promising but challenging.

• Existing FPGA-based ViT accelerators mainly rely on temporal architectures, which process different operators by reusing the same hardware blocks and suffer from extensive memory access overhead.
• Pipelined architectures, either coarse-grained or fine-grained, unroll the ViT computation spatially for memory access efficiency.
• However, they usually suffer from significant hardware resource constraints and pipeline bubbles induced by the global computation dependency of ViT.