Parallel Continuous Chain-of-Thought with Jacobi Iteration

Understand

Continuous chain-of-thought has been shown to be effective in saving reasoning tokens for large language models.

• By reasoning with continuous latent thought tokens, continuous CoT is able to perform implicit reasoning in a compact manner.
• However, the sequential dependencies between latent thought tokens spoil parallel training, leading to long training time.
• In this paper, we propose Parallel Continuous Chain-of-Thought (PCCoT), which performs Jacobi iteration on the latent thought tokens, updating them iteratively in parallel instead of sequentially and thus improving both training and inference efficiency of continuous CoT.