Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators

Understand

Recent text-to-video generation approaches rely on computationally heavy training and require large-scale video datasets.

• In this paper, we introduce a new task of zero-shot text-to-video generation and propose a low-cost approach (without any training or optimization) by leveraging the power of existing text-to-image synthesis methods (e.g., Stable Diffusion), making them suitable for the video domain.
• Our key modifications include (i) enriching the latent codes of the generated frames with motion dynamics to keep the global scene and the background time consistent; and (ii) reprogramming frame-level self-attention using a new cross-frame attention of each frame on the first frame, to preserve the context, appearance, and identity of the foreground object.
• Experiments show that this leads to low overhead, yet high-quality and remarkably consistent video generation.