VidTwin: Video VAE with Decoupled Structure and Dynamics

Understand

Recent advancements in video autoencoders (Video AEs) have significantly improved the quality and efficiency of video generation.

• In this paper, we propose a novel and compact video autoencoder, VidTwin, that decouples video into two distinct latent spaces: Structure latent vectors, which capture overall content and global movement, and Dynamics latent vectors, which represent fine-grained details and rapid movements.
• Specifically, our approach leverages an Encoder-Decoder backbone, augmented with two submodules for extracting these latent spaces, respectively.
• The first submodule employs a Q-Former to extract low-frequency motion trends, followed by downsampling blocks to remove redundant content details.