Toward Sim-to-Real Directional Semantic Grasping

Understand

We address the problem of directional semantic grasping, that is, grasping a specific object from a specific direction.

• We approach the problem using deep reinforcement learning via a double deep Q-network (DDQN) that learns to map downsampled RGB input images from a wrist-mounted camera to Q-values, which are then translated into Cartesian robot control commands via the cross-entropy method (CEM).
• The network is learned entirely on simulated data generated by a custom robot simulator that models both physical reality (contacts) and perceptual quality (high-quality rendering).
• The reality gap is bridged using domain randomization.