CLEAR: Conv-Like Linearization Revs Pre-Trained Diffusion Transformers Up

Understand

Diffusion Transformers (DiT) have become a leading architecture in image generation.

• However, the quadratic complexity of attention mechanisms, which are responsible for modeling token-wise relationships, results in significant latency when generating high-resolution images.
• To address this issue, we aim at a linear attention mechanism in this paper that reduces the complexity of pre-trained DiTs to linear.
• We begin our exploration with a comprehensive summary of existing efficient attention mechanisms and identify four key factors crucial for successful linearization of pre-trained DiTs: locality, formulation consistency, high-rank attention maps, and feature integrity.