Spherical World-Locking for Audio-Visual Localization in Egocentric Videos

Understand

Egocentric videos provide comprehensive contexts for user and scene understanding, spanning multisensory perception to behavioral interaction.

• We propose Spherical World-Locking (SWL) as a general framework for egocentric scene representation, which implicitly transforms multisensory streams with respect to measurements of head orientation.
• Compared to conventional head-locked egocentric representations with a 2D planar field-of-view, SWL effectively offsets challenges posed by self-motion, allowing for improved spatial synchronization between input modalities.
• Using a set of multisensory embeddings on a worldlocked sphere, we design a unified encoder-decoder transformer architecture that preserves the spherical structure of the scene representation, without requiring expensive projections between image and world coordinate systems.