PLAICraft: Large-Scale Time-Aligned Vision-Speech-Action Dataset for Embodied AI

Understand

Advances in deep generative modeling have made it increasingly plausible to train human-level embodied agents.

• Yet progress has been limited by the absence of large-scale, real-time, multi-modal, and socially interactive datasets that reflect the sensory-motor complexity of natural environments.
• To address this, we present PLAICraft, a novel data collection platform and dataset capturing multiplayer Minecraft interactions across five time-aligned modalities: video, game output audio, microphone input audio, mouse, and keyboard actions.
• Each modality is logged with millisecond time precision, enabling the study of synchronous, embodied behaviour in a rich, open-ended world.