2022

Large Language Models are Pretty Good Zero-Shot Video Game Bug Detectors

Taesiri, Mohammad Reza, Macklon, Finlay, Wang, Yihe et al.

Understand

Video game testing requires game-specific knowledge as well as common sense reasoning about the events in the game.

While AI-driven agents can satisfy the first requirement, it is not yet possible to meet the second requirement automatically.
Therefore, video game testing often still relies on manual testing, and human testers are required to play the game thoroughly to detect bugs.
As a result, it is challenging to fully automate game testing.

Built on

Language models are few-shot learners
Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J. D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al. 2020 · 1901
Earlier work this paper cites.
Dota 2 with large scale deep reinforcement learning
Original
Berner, C.; Brockman, G.; Chan, B.; Cheung, V.; Dębiak, P.; Dennison, C.; Farhi, D.; Fischer, Q.; Hashme, S.; Hesse, C.; et al. 2019 · 1912
Earlier work this paper cites.
Scaling laws for neural language models
Original
Kaplan, J.; McCandlish, S.; Henighan, T.; Brown, T. B.; Chess, B.; Child, R.; Gray, S.; Radford, A.; Wu, J.; and Amodei, D. 2020 · 2001
Earlier work this paper cites.
Game physics engine development
Millington, I. 2007 · 2007
Earlier work this paper cites.
Wilcoxon signed-rank test
Woolson, R. F. 2007 · 2007
Earlier work this paper cites.
It’s not just size that matters: Small language models are also few-shot learners
Original
Schick, T.; and Schütze, H. 2020 · 2009
Earlier work this paper cites.
Making pre-trained language models better few-shot learners
Original
Gao, T.; Fisch, A.; and Chen, D. 2020 · 2012
Earlier work this paper cites.
Distilling the knowledge in a neural network
Original
Hinton, G.; Vinyals, O.; Dean, J.; et al. 2015 · 2015
Earlier work this paper cites.
Structured pruning of deep convolutional neural networks
Anwar, S.; Hwang, K.; and Sung, W. 2017 · 2017
Earlier work this paper cites.
Attention is all you need
Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017 · 2017
Earlier work this paper cites.
Bert: Pre-training of deep bidirectional transformers for language understanding
Original
Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2018 · 2018
Earlier work this paper cites.
The natural language decathlon: Multitask learning as question answering
Original
McCann, B.; Keskar, N. S.; Xiong, C.; and Socher, R. 2018 · 2018
Earlier work this paper cites.
How is video game development different from software development in open source?
Pascarella, L.; Palomba, F.; Di Penta, M.; and Bacchelli, A. 2018 · 2018
Earlier work this paper cites.
Faulty reward functions in the wild
Clark, J.; and Amodei, D. 2019 · 2019
Earlier work this paper cites.
Deep learning for video game playing
Justesen, N.; Bontrager, P.; Togelius, J.; and Risi, S. 2019 · 2019
Earlier work this paper cites.
Monster carlo 2: Integrating learning and tree search for machine playtesting
Keehl, O.; and Smith, A. M. 2019 · 2019
Earlier work this paper cites.
Language models are unsupervised multitask learners
Radford, A.; Wu, J.; Child, R.; Luan, D.; Amodei, D.; Sutskever, I.; et al. 2019 · 2019
Earlier work this paper cites.

Similar

Grandmaster level in StarCraft II using multi-agent reinforcement learning
Vinyals, O.; Babuschkin, I.; Czarnecki, W. M.; Mathieu, M.; Dudzik, A.; Chung, J.; Choi, D. H.; Powell, R.; Ewalds, T.; Georgiev, P.; et al. 2019 · 2019
Cited alongside, same era.
Retrieving videogame moments with natural language queries
Zhang, X.; and Smith, A. M. 2019 · 2019
Cited alongside, same era.
Wuji: Automatic online combat game testing using evolutionary deep reinforcement learning
Zheng, Y.; Xie, X.; Su, T.; Ma, L.; Hao, J.; Meng, Z.; Liu, Y.; Shen, R.; Chen, Y.; and Fan, C. 2019 · 2019
Cited alongside, same era.
Emergent Tool Use From Multi-Agent Autocurricula
Baker, B.; Kanitscheider, I.; Markov, T.; Wu, Y.; Powell, G.; McGrew, B.; and Mordatch, I. 2020 · 2020
Cited alongside, same era.
Using deep convolutional neural networks to detect rendered glitches in video games
Ling, C.; Tollmar, K.; and Gisslén, L. 2020 · 2020
Cited alongside, same era.
Predicting game difficulty and churn without players
Roohi, S.; Relas, A.; Takatalo, J.; Heiskanen, H.; and Hämäläinen, P. 2020 · 2020
Cited alongside, same era.
A video game testing method utilizing deep learning
Taesiri, M. R.; Habibi, M.; and Fazli, M. A. 2020 · 2020
Cited alongside, same era.
Evaluating large language models trained on code
Original
Chen, M.; Tworek, J.; Jun, H.; Yuan, Q.; Pinto, H. P. d. O.; Kaplan, J.; Edwards, H.; Burda, Y.; Joseph, N.; Brockman, G.; et al. 2021 · 2021
Cited alongside, same era.
Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing
Original
Liu, P.; Yuan, W.; Fu, J.; Jiang, Z.; Hayashi, H.; and Neubig, G. 2021 · 2021
Cited alongside, same era.
Clipcap: Clip prefix for image captioning
Original
Mokady, R.; Hertz, A.; and Bermano, A. H. 2021 · 2021
Cited alongside, same era.
A survey of video game testing
Politowski, C.; Petrillo, F.; and Guéhéneuc, Y.-G. 2021 · 2021
Cited alongside, same era.
Learning transferable visual models from natural language supervision
Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021 · 2021
Cited alongside, same era.

Then

Netcut: Real-time dnn inference using layer removal
Zandigohar, M.; Erdoğmuş, D.; and Schirner, G. 2021 · 2021
Later among the works it cites.
Do as I can, not as I say: Grounding language in robotic affordances
Original
Ahn, M.; Brohan, A.; Brown, N.; Chebotar, Y.; Cortes, O.; David, B.; Finn, C.; Gopalakrishnan, K.; Hausman, K.; Herzog, A.; et al. 2022 · 2022
Closest in time.
Palm: Scaling language modeling with pathways
Original
Chowdhery, A.; Narang, S.; Devlin, J.; Bosma, M.; Mishra, G.; Roberts, A.; Barham, P.; Chung, H. W.; Sutton, C.; Gehrmann, S.; et al. 2022 · 2022
Closest in time.
LLM. int8 (): 8-bit Matrix Multiplication for Transformers at Scale
Original
Dettmers, T.; Lewis, M.; Belkada, Y.; and Zettlemoyer, L. 2022 · 2022
Closest in time.
Language Model Cascades
Original
Dohan, D.; Xu, W.; Lewkowycz, A.; Austin, J.; Bieber, D.; Lopes, R. G.; Wu, Y.; Michalewski, H.; Saurous, R. A.; Sohl-dickstein, J.; et al. 2022 · 2022
Closest in time.
Jigsaw: large language models meet program synthesis
Jain, N.; Vaidyanath, S.; Iyer, A.; Natarajan, N.; Parthasarathy, S.; Rajamani, S.; and Sharma, R. 2022 · 2022
Closest in time.
Large Language Models are Zero-Shot Reasoners
Kojima, T.; Gu, S. S.; Reid, M.; Matsuo, Y.; and Iwasawa, Y. 2022 · 2022
Closest in time.
Automatically Detecting Visual Bugs in HTML5< canvas> Games
Original
Macklon, F.; Taesiri, M. R.; Viggiato, M.; Antoszko, S.; Romanova, N.; Paas, D.; and Bezemer, C.-P. 2022 · 2022
Closest in time.
Training language models to follow instructions with human feedback
Original
Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright, C. L.; Mishkin, P.; Zhang, C.; Agarwal, S.; Slama, K.; Ray, A.; et al. 2022 · 2022
Closest in time.
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Original
Srivastava, A.; Rastogi, A.; Rao, A.; Shoeb, A. A. M.; Abid, A.; Fisch, A.; Brown, A. R.; Santoro, A.; Gupta, A.; Garriga-Alonso, A.; et al. 2022 · 2022
Closest in time.
CLIP meets GamePhysics: Towards bug identification in gameplay videos using zero-shot transfer learning
Taesiri, M. R.; Macklon, F.; and Bezemer, C.-P. 2022 · 2022
Closest in time.
ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic
Tewel, Y.; Shalev, Y.; Schwartz, I.; and Wolf, L. 2022 · 2022
Closest in time.
Lamda: Language models for dialog applications
Original
Thoppilan, R.; De Freitas, D.; Hall, J.; Shazeer, N.; Kulshreshtha, A.; Cheng, H.-T.; Jin, A.; Bos, T.; Baker, L.; Du, Y.; et al. 2022 · 2022
Closest in time.
Using natural language processing techniques to improve manual test case descriptions
Viggiato, M.; Paas, D.; Buzon, C.; and Bezemer, C.-P. 2022b · 2022
Closest in time.
OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework
Wang, P.; Yang, A.; Men, R.; Lin, J.; Bai, S.; Li, Z.; Ma, J.; Zhou, C.; Zhou, J.; and Yang, H. 2022 · 2022
Closest in time.
Chain of thought prompting elicits reasoning in large language models
Original
Wei, J.; Wang, X.; Schuurmans, D.; Bosma, M.; Chi, E.; Le, Q.; and Zhou, D. 2022 · 2022
Closest in time.
Opt: Open pre-trained transformer language models
Original
Zhang, S.; Roller, S.; Goyal, N.; Artetxe, M.; Chen, M.; Chen, S.; Dewan, C.; Diab, M.; Li, X.; Lin, X. V.; et al. 2022 · 2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…