ME-ViT: A Single-Load Memory-Efficient FPGA Accelerator for Vision Transformers

Understand

Vision Transformers (ViTs) have emerged as a state-of-the-art solution for object classification tasks.

• However, their computational demands and high parameter count make them unsuitable for real-time inference, prompting the need for efficient hardware implementations.
• Existing hardware accelerators for ViTs suffer from frequent off-chip memory access, restricting the achievable throughput by memory bandwidth.
• In devices with a high compute-to-communication ratio (e.g., edge FPGAs with limited bandwidth), off-chip memory access imposes a severe bottleneck on overall throughput.