Re$^3$Sim: Generating High-Fidelity Simulation Data via 3D-Photorealistic Real-to-Sim for Robotic Manipulation

Understand

Real-world data collection for robotics is costly and resource-intensive, requiring skilled operators and expensive hardware.

• Simulations offer a scalable alternative but often fail to achieve sim-to-real generalization due to geometric and visual gaps.
• To address these challenges, we propose a 3D-photorealistic real-to-sim system, namely, RE$^3$SIM, addressing geometric and visual sim-to-real gaps.
• RE$^3$SIM employs advanced 3D reconstruction and neural rendering techniques to faithfully recreate real-world scenarios, enabling real-time rendering of simulated cross-view cameras within a physics-based simulator.