Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodality

Understand

In Transformer architectures, tokens\textemdash discrete units derived from raw data\textemdash are formed by segmenting inputs into fixed-length chunks.

• Each token is then mapped to an embedding, enabling parallel attention computations while preserving the input's essential information.
• Due to the quadratic computational complexity of transformer self-attention mechanisms, token reduction has primarily been used as an efficiency strategy.
• This is especially true in single vision and language domains, where it helps balance computational costs, memory usage, and inference latency.