Latent Knowledge-Guided Video Diffusion for Scientific Phenomena Generation from a Single Initial Frame

Understand

Video diffusion models have achieved impressive results in natural scene generation, yet they struggle to generalize to scientific phenomena such as fluid simulations and meteorological processes, where underlying dynamics are governed by scientific laws.

• These tasks pose unique challenges, including severe domain gaps, limited training data, and the lack of descriptive language annotations.
• To handle this dilemma, we extracted the latent scientific phenomena knowledge and further proposed a fresh framework that teaches video diffusion models to generate scientific phenomena from a single initial frame.
• Particularly, static knowledge is extracted via pre-trained masked autoencoders, while dynamic knowledge is derived from pre-trained optical flow prediction.