Cross-Task Generalization Between Understanding and Generation in Unified Vision-Language Models: A Controlled Study

Understand

Unified vision-language models (VLMs) aim to support both visual understanding and generation within a single framework, but it remains unclear when mixed training benefits both capabilities and when it introduces conflicts.

• This paper presents a controlled empirical study of cross-task generalization between understanding and generation in unified VLMs.
• We construct two controllable image-text benchmarks, SmartWatch and modified CelebA, with paired VQA, captioning, and text-to-image generation tasks, and evaluate multiple LLM-based unified architectures built from SigLIP and VQ-VAE visual spaces.
• Our experiments show that mixed understanding-generation training can improve both tasks over task-specific training, but the benefit depends strongly on the relation between vision input and output spaces.