SAM-RL: Sensing-Aware Model-Based Reinforcement Learning via Differentiable Physics-Based Simulation and Rendering

Understand

Model-based reinforcement learning (MBRL) is recognized with the potential to be significantly more sample-efficient than model-free RL.

• How an accurate model can be developed automatically and efficiently from raw sensory inputs (such as images), especially for complex environments and tasks, is a challenging problem that hinders the broad application of MBRL in the real world.
• In this work, we propose a sensing-aware model-based reinforcement learning system called SAM-RL.
• Leveraging the differentiable physics-based simulation and rendering, SAM-RL automatically updates the model by comparing rendered images with real raw images and produces the policy efficiently.