Out of Sight, Still in Mind: Reasoning and Planning about Unobserved Objects with Video Tracking Enabled Memory Models

Understand

Robots need to have a memory of previously observed, but currently occluded objects to work reliably in realistic environments.

• We investigate the problem of encoding object-oriented memory into a multi-object manipulation reasoning and planning framework.
• We propose DOOM and LOOM, which leverage transformer relational dynamics to encode the history of trajectories given partial-view point clouds and an object discovery and tracking engine.
• Our approaches can perform multiple challenging tasks including reasoning with occluded objects, novel objects appearance, and object reappearance.