History-Aware Visuomotor Policy Learning via Point Tracking

Understand

Many manipulation tasks require memory beyond the current observation, yet most visuomotor policies rely on the Markov assumption and thus struggle with repeated states or long-horizon dependencies.

• Existing methods attempt to extend observation horizons but remain insufficient for diverse memory requirements.
• To this end, we propose an object-centric history representation based on point tracking, which abstracts past observations into a compact and structured form that retains only essential task-relevant information.
• Tracked points are encoded and aggregated at the object level, yielding a compact history representation that can be seamlessly integrated into various visuomotor policies.