CoPhy: Counterfactual Learning of Physical Dynamics

Understand

Understanding causes and effects in mechanical systems is an essential component of reasoning in the physical world.

• This work poses a new problem of counterfactual learning of object mechanics from visual input.
• We develop the CoPhy benchmark to assess the capacity of the state-of-the-art models for causal physical reasoning in a synthetic 3D environment and propose a model for learning the physical dynamics in a counterfactual setting.
• Having observed a mechanical experiment that involves, for example, a falling tower of blocks, a set of bouncing balls or colliding objects, we learn to predict how its outcome is affected by an arbitrary intervention on its initial conditions, such as displacing one of the objects in the scene.