DualToken: Towards Unifying Visual Understanding and Generation with Dual Visual Vocabularies

Understand

The differing representation spaces required for visual understanding and generation pose a challenge in unifying them within the autoregressive paradigm of large language models.

• A vision tokenizer trained for reconstruction excels at capturing low-level visual appearance, making it well-suited for visual generation but lacking high-level semantic representations for understanding tasks.
• Conversely, a vision encoder trained via contrastive learning aligns well with language but struggles to decode back into the pixel space for generation tasks.
• To bridge this gap, we propose DualToken, a method that unifies representations for both understanding and generation within a single tokenizer.