Learning Modal-Mixed Chain-of-Thought Reasoning with Latent Embeddings

Understand

We study how to extend chain-of-thought (CoT) beyond language to better handle multimodal reasoning.

• While CoT helps LLMs and VLMs articulate intermediate steps, its text-only form often fails on vision-intensive problems where key intermediate states are inherently visual.
• We introduce modal-mixed CoT, which interleaves textual tokens with compact visual sketches represented as latent embeddings.
• To bridge the modality gap without eroding the original knowledge and capability of the VLM, we use the VLM itself as an encoder and train the language backbone to reconstruct its own intermediate vision embeddings, to guarantee the semantic alignment of the visual latent space.