Rethinking Progression of Memory State in Robotic Manipulation: An Object-Centric Perspective

Understand

As embodied agents operate in increasingly complex environments, the ability to perceive, track, and reason about individual object instances over time becomes essential, especially in tasks requiring sequenced interactions with visually similar objects.

• In these non-Markovian settings, key decision cues are often hidden in object-specific histories rather than the current scene.
• Without persistent memory of prior interactions (what has been interacted with, where it has been, or how it has changed) visuomotor policies may fail, repeat past actions, or overlook completed ones.
• To surface this challenge, we introduce LIBERO-Mem, a non-Markovian task suite for stress-testing robotic manipulation under object-level partial observability.