Zero-TPrune: Zero-Shot Token Pruning through Leveraging of the Attention Graph in Pre-Trained Transformers

Understand

Deployment of Transformer models on edge devices is becoming increasingly challenging due to the exponentially growing inference cost that scales quadratically with the number of tokens in the input sequence.

• Token pruning is an emerging solution to address this challenge due to its ease of deployment on various Transformer backbones.
• However, most token pruning methods require computationally expensive fine-tuning, which is undesirable in many edge deployment cases.
• In this work, we propose Zero-TPrune, the first zero-shot method that considers both the importance and similarity of tokens in performing token pruning.