Extracting Replayable Interactions from Videos of Mobile App Usage

Understand

Screen recordings of mobile apps are a popular and readily available way for users to share how they interact with apps, such as in online tutorial videos, user reviews, or as attachments in bug reports.

• Unfortunately, both people and systems can find it difficult to reproduce touch-driven interactions from video pixel data alone.
• In this paper, we introduce an approach to extract and replay user interactions in videos of mobile apps, using only pixel information in video frames.
• To identify interactions, we apply heuristic-based image processing and convolutional deep learning to segment screen recordings, classify the interaction in each segment, and locate the interaction point.