2022

Automatically Detecting Visual Bugs in HTML5 <canvas> Games

Macklon, Finlay, Taesiri, Mohammad Reza, Viggiato, Markos et al.

Understand

The HTML5 <canvas> is used to display high quality graphics in web applications such as web games (i.e., <canvas> games).

However, automatically testing <canvas> games is not possible with existing web testing techniques and tools, and manual testing is laborious.
Many widely used web testing tools rely on the Document Object Model (DOM) to drive web test automation, but the contents of the <canvas> are not represented in the DOM.
The main alternative approach, snapshot testing, involves comparing oracle snapshot images with test-time snapshot images using an image similarity metric to catch visual bugs, i.e., bugs in the graphics of the web application.

Built on

“On a test of whether one of two random variables is stochastically larger than the other”
Henry Mann and Donald Whitney · 1947
Earlier work this paper cites.
“Dominance statistics: Ordinal analyses to answer ordinal questions.”
Norman Cliff · 1993
Earlier work this paper cites.
“Image quality measures and their performance”
Ahmet Eskicioglu and Paul Fisher · 1995
Earlier work this paper cites.
“A universal image quality index”
Zhou Wang and Alan Bovik · 2002
Earlier work this paper cites.
“Image quality assessment: from error visibility to structural similarity”
Zhou Wang, Alan Bovik, Hamid Sheikh and Eero Simoncelli · 2004
Earlier work this paper cites.
“Exploring methods for evaluating group differences on the NSSE and other surveys: Are the t-test and Cohen’sd indices the most appropriate choices”
Jeanine Romano et al · 2006
Earlier work this paper cites.
“A Framework for the Semi-Automatic Testing of Video Games.”
Alfredo Nantes, Ross Brown and Frederic Maire · 2008
Earlier work this paper cites.
“What went wrong? A survey of problems in game development”
Fábio Petrillo, Marcelo Pimenta, Francisco Trindade and Carlos Dietrich · 2009
Earlier work this paper cites.
“A temporal perspective of the computer game development process”
Patrick Stacey and Joe Nandhakumar · 2009
Earlier work this paper cites.
“What went wrong: a taxonomy of video game bugs”
Chris Lewis, Jim Whitehead and Noah Wardrip-Fruin · 2010
Earlier work this paper cites.
“Visual testing of Graphical User Interfaces: An exploratory study towards systematic definitions and approaches”
Ayman Issa, Jonathan Sillito and Vahid Garousi · 2012
Earlier work this paper cites.
“WebGL: up and running”
Tony Parisi · 2012
Earlier work this paper cites.
“HTML5 canvas: native interactivity and animation for the web”
Steve Fulton and Jeff Fulton · 2013
Earlier work this paper cites.
“Cowboys, ankle sprains, and keepers of quality: How is video game development different from software development?”
Emerson Murphy-Hill, Thomas Zimmermann and Nachiappan Nagappan · 2014
Earlier work this paper cites.
“Programming 3D Applications with HTML5 and WebGL: 3D Animation and Visualization for Web Pages”
Tony Parisi · 2014
Earlier work this paper cites.
“Comparison of physics frameworks for WebGL-based game engine”
Resa Yogya and Raymond Kosala · 2014
Earlier work this paper cites.
“Deep residual learning for image recognition. arXiv 2015”
Original
Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun · 2015
Earlier work this paper cites.
“Moving real exergaming engines on the web: the webFitForAll case study in an active and healthy ageing living lab environment”
Evdokimos Konstantinidis, Giorgos Bamparopoulos and Panagiotis Bamidis · 2016
Earlier work this paper cites.

Similar

“Unity application testing automation with appium and image recognition”
Maxim Mozgovoy and Evgeny Pyshkin · 2017
Cited alongside, same era.
“Malicious software classification using transfer learning of resnet-50 deep neural network”
Edmar Rezende et al · 2017
Cited alongside, same era.
“Web canvas testing through visual inference”
Mohammad Bajammal and Ali Mesbah · 2018
Cited alongside, same era.
“Player experience extraction from gameplay video”
Zijin Luo, Matthew Guzdial, Nicholas Liao and Mark Riedl · 2018
Cited alongside, same era.
“The unreasonable effectiveness of deep features as a perceptual metric”
Richard Zhang et al · 2018
Cited alongside, same era.
“Making cnns for video parsing accessible: event extraction from dota2 gameplay video using transfer, zero-shot, and network pruning”
Zijin Luo, Matthew Guzdial and Mark Riedl · 2019
Cited alongside, same era.
“Improving random GUI testing with image-based widget detection”
Thomas White, Gordon Fraser and Guy Brown · 2019
Cited alongside, same era.
“Investigating Web3D topics on StackOverflow: a preliminary study of WebGL and Three.js”
Farag Almansoury, Sègla Kpodjedo and Ghizlane Boussaidi · 2020
Cited alongside, same era.
“A Survey on the Use of Computer Vision to Improve Software Engineering Tasks”
Mohammad Bajammal, Andrea Stocco, Davood Mazinanian and Ali Mesbah · 2020
Cited alongside, same era.
“Unblind your apps: Predicting natural-language labels for mobile gui components by deep learning”
Jieshan Chen et al · 2020
Cited alongside, same era.
“Automating Artifact Detection in Video Games”
Original
Parmida Davarmanesh et al · 2020
Cited alongside, same era.
“Synthesizing Retro Game Screenshot Datasets for Sprite Detection.”
Chanha Kim, Jaden Kim and Joseph Osborn · 2020
Cited alongside, same era.

Then

“Page Segmentation using Visual Adjacency Analysis”
Original
Mohammad Bajammal and Ali Mesbah · 2021
Later among the works it cites.
“Semantic Web Accessibility Testing via Hierarchical Visual Analysis”
Mohammad Bajammal and Ali Mesbah · 2021
Later among the works it cites.
“GLIB: towards automated test oracle for graphically-rich applications”
Ke Chen et al · 2021
Later among the works it cites.
“Under the Skin of Foundation NFT Auctions”
Original
MohammadAmin Fazli, Ali Owfi and Mohammad Taesiri · 2021
Later among the works it cites.
“An empirical study of Q&A websites for game developers”
Arthur Kamienski and Cor-Paul Bezemer · 2021
Later among the works it cites.
“Style-Guided Web Application Exploration”
Original
Davood Mazinanian, Mohammad Bajammal and Ali Mesbah · 2021
Later among the works it cites.
“Playco, Global Leader in Instant Games, Acquires Goodboy Digital, Creators of PixiJS, the Number One HTML5 Game Engine.” Last accessed 5 May 2022., 2021
Playco · 2021
Later among the works it cites.
“AI-based Test Automation: A Grey Literature Analysis”
Filippo Ricca, Alessandro Marchetto and Andrea Stocco · 2021
Later among the works it cites.
“Marionette: Self-supervised sprite learning”
Dmitriy Smirnov et al · 2021
Later among the works it cites.
“Fragment-Based Test Generation For Web Apps”
Original
Rahulkrishna Yandrapally and Ali Mesbah · 2021
Later among the works it cites.
“An empirical study of GUI widget detection for industrial mobile games”
Jiaming Ye et al · 2021
Later among the works it cites.
“Forum - Frameworks” Last accessed 4 May 2022., 2022
HTML5Game Devs · 2022
Closest in time.
“A Taxonomy of HTML5 Canvas Bugs”
Original
Finlay Macklon et al · 2022
Closest in time.
“CLIP meets GamePhysics: Towards bug identification in gameplay videos using zero-shot transfer learning”
Mohammad Taesiri, Finlay Macklon and Cor-Paul Bezemer · 2022
Closest in time.
“Identifying Similar Test Cases That Are Specified in Natural Language”
Markos Viggiato, Dale Paas, Chris Buzon and Cor-Paul Bezemer · 2022
Closest in time.
“Using Natural Language Processing Techniques to Improve Manual Test Case Descriptions”
Markos Viggiato, Dale Paas, Chris Buzon and Cor-Paul Bezemer · 2022
Closest in time.
“Learning-Replay Based Automated Robotic Testing for Mobile App”
Feng Xue, Junsheng Wu and Tao Zhang · 2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…