LLM-grounded Video Diffusion Models

Understand

Text-conditioned diffusion models have emerged as a promising tool for neural video generation.

• However, current models still struggle with intricate spatiotemporal prompts and often generate restricted or incorrect motion.
• To address these limitations, we introduce LLM-grounded Video Diffusion (LVD).
• Instead of directly generating videos from the text inputs, LVD first leverages a large language model (LLM) to generate dynamic scene layouts based on the text inputs and subsequently uses the generated layouts to guide a diffusion model for video generation.