A LLM Benchmark based on the Minecraft Builder Dialog Agent Task

Craftassist: A framework for dialogue-enabled interactive agents

Original

Jonathan Gray, Kavya Srinet, Yacine Jernite, Haonan Yu, Zhuoyuan Chen, Demi Guo, Siddharth Goyal, C Lawrence Zitnick, and Arthur Szlam. 2019 · 1907

Earlier work this paper cites.

Why build an assistant in minecraft?

Original

Arthur Szlam, Jonathan Gray, Kavya Srinet, Yacine Jernite, Armand Joulin, Gabriel Synnaeve, Douwe Kiela, Haonan Yu, Zhuoyuan Chen, Siddharth Goyal, et al. 2019 · 1907

Earlier work this paper cites.

Understanding natural language

Terry Winograd. 1972 · 1972

Earlier work this paper cites.

Towards ai-complete question answering: A set of prerequisite toy tasks

Original

Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart Van Merriënboer, Armand Joulin, and Tomas Mikolov. 2015 · 2015

Earlier work this paper cites.

The malmo platform for artificial intelligence experimentation

Matthew Johnson, Katja Hofmann, Tim Hutton, and David Bignell. 2016 · 2016

Earlier work this paper cites.

Collaborative dialogue in minecraft

Anjali Narayan-Chen, Prashant Jayannavar, and Julia Hockenmaier. 2019 · 2019

Earlier work this paper cites.

Learning to speak and act in a fantasy text adventure game

Jack Urbanek, Angela Fan, Siddharth Karamcheti, Emily Dinan Saachi Jain, Samuel Humeau, Tim Rocktäschel, Douwe Kiela, Arthur Szlam, and Jason Weston. 2019 · 2019

Earlier work this paper cites.

Learning to execute instructions in a minecraft dialogue

Prashant Jayannavar, Anjali Narayan-Chen, and Julia Hockenmaier. 2020 · 2020

Cited alongside, same era.

Mc-saar-instruct: a platform for minecraft instruction giving agents

Arne Köhn, Julia Wichlacz, Christine Schäfer, Alvaro Torralba, Jörg Hoffmann, and Alexander Koller. 2020 · 2020

Cited alongside, same era.

Gamification platform for collecting task-oriented dialogue data

Haruna Ogawa, Hitoshi Nishikawa, Takenobu Tokunaga, and Hikaru Yokono. 2020 · 2020

Cited alongside, same era.

Interactive grounded language understanding in a collaborative environment: Iglu 2021

Julia Kiseleva, Ziming Li, Mohammad Aliannejadi, Shrestha Mohanty, Maartje ter Hoeve, Mikhail Burtsev, Alexey Skrynnik, Artem Zholus, Aleksandr Panov, Kavya Srinet, et al. 2022 · 2021

Cited alongside, same era.

Dungeons and dragons as a dialog challenge for artificial intelligence

Original

Chris Callison-Burch, Gaurav Singh Tomar, Lara J Martin, Daphne Ippolito, Suma Bailis, and David Reitter. 2022 · 2022

Cited alongside, same era.

Stepgame: A new benchmark for robust multi-hop spatial reasoning in texts

Zhengxiang Shi, Qiang Zhang, and Aldo Lipani. 2022 · 2022

Later among the works it cites.

Beyond the imitation game: Quantifying and extrapolating the capabilities of language models

Original

Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. 2022 · 2022

Later among the works it cites.

Chain-of-thought prompting elicits reasoning in large language models

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022 · 2022

Later among the works it cites.

A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity

Original

Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. 2023 · 2023

Later among the works it cites.

Smartplay: A benchmark for llms as intelligent agents