What Are You Doing? A Closer Look at Controllable Human Video Generation

Understand

High-quality benchmarks are crucial for driving progress in machine learning research.

• However, despite the growing interest in video generation, there is no comprehensive dataset to evaluate human synthesis.
• Humans can perform a wide variety of actions and interactions, but existing datasets, like TikTok, TED-Talks, and HumanVid, lack the diversity and complexity to fully capture the capabilities of video generation models.
• We close this gap by introducing 'What Are You Doing?' (WYD): a new benchmark for fine-grained evaluation of controllable image-to-video generation of humans.