Structure and Content-Guided Video Synthesis with Diffusion Models

Understand

Text-guided generative diffusion models unlock powerful image creation and editing tools.

• While these have been extended to video generation, current approaches that edit the content of existing footage while retaining structure require expensive re-training for every input or rely on error-prone propagation of image edits across frames.
• In this work, we present a structure and content-guided video diffusion model that edits videos based on visual or textual descriptions of the desired output.
• Conflicts between user-provided content edits and structure representations occur due to insufficient disentanglement between the two aspects.