Uncertainty-aware State Space Transformer for Egocentric 3D Hand Trajectory Forecasting

Understand

Hand trajectory forecasting from egocentric views is crucial for enabling a prompt understanding of human intentions when interacting with AR/VR systems.

• However, existing methods handle this problem in a 2D image space which is inadequate for 3D real-world applications.
• In this paper, we set up an egocentric 3D hand trajectory forecasting task that aims to predict hand trajectories in a 3D space from early observed RGB videos in a first-person view.
• To fulfill this goal, we propose an uncertainty-aware state space Transformer (USST) that takes the merits of the attention mechanism and aleatoric uncertainty within the framework of the classical state-space model.