A Control-Centric Benchmark for Video Prediction

Understand

Video is a promising source of knowledge for embodied agents to learn models of the world's dynamics.

• Large deep networks have become increasingly effective at modeling complex video data in a self-supervised manner, as evaluated by metrics based on human perceptual similarity or pixel-wise comparison.
• However, it remains unclear whether current metrics are accurate indicators of performance on downstream tasks.
• We find empirically that for planning robotic manipulation, existing metrics can be unreliable at predicting execution success.