Playing for Data: Ground Truth from Computer Games

Understand

Recent progress in computer vision has been driven by high-capacity models trained on large datasets.

• Unfortunately, creating large datasets with pixel-level labels has been extremely costly due to the amount of human effort required.
• In this paper, we present an approach to rapidly creating pixel-accurate semantic label maps for images extracted from modern computer games.
• Although the source code and the internal operation of commercial games are inaccessible, we show that associations between image patches can be reconstructed from the communication between the game and the graphics hardware.