When Less is Enough: Adaptive Token Reduction for Efficient Image Representation

Understand

Vision encoders typically generate a large number of visual tokens, providing information-rich representations but significantly increasing computational demands.

• This raises the question of whether all generated tokens are equally valuable or if some of them can be discarded to reduce computational costs without compromising quality.
• In this paper, we introduce a new method for determining feature utility based on the idea that less valuable features can be reconstructed from more valuable ones.
• We implement this concept by integrating an autoencoder with a Gumbel-Softmax selection mechanism, that allows identifying and retaining only the most informative visual tokens.