Tell Me What Happened: Unifying Text-guided Video Completion via Multimodal Masked Video Generation

Understand

Generating a video given the first several static frames is challenging as it anticipates reasonable future frames with temporal coherence.

• Besides video prediction, the ability to rewind from the last frame or infilling between the head and tail is also crucial, but they have rarely been explored for video completion.
• Since there could be different outcomes from the hints of just a few frames, a system that can follow natural language to perform video completion may significantly improve controllability.
• Inspired by this, we introduce a novel task, text-guided video completion (TVC), which requests the model to generate a video from partial frames guided by an instruction.