VISA: VLM-Guided Instance Semantic Auditing for 3D Occupancy World Models

Understand

Semantic 3D occupancy provides a voxelized world state for autonomous driving and robot decision making, but object and rare-class errors can affect free-space interpretation, collision checking, and temporal state propagation.

• We show that a common VLM strategy, aligning 3D voxel or object features with crop-caption embeddings, improves text-space similarity without reliably improving closed-set occupancy mIoU.
• Motivated by this mismatch, we propose VISA, a training-time semantic auditing approach for existing occupancy world models.
• VISA queries an offline VLM on a representative crop of each physical object instance, obtains a structured audit with class hypotheses, plausible confusions, reliability, attributes, and evidence, and propagates it along the object track.