Fetching the paper…

Reading the bibliography…

2023

Evaluating Language-Model Agents on Realistic Autonomous Tasks

Kinniment, Megan, Sato, Lucas Jun Koba, Du, Haoxing et al.

Original
PDF
Post on X

Understand

In this report, we explore the ability of language model agents to acquire resources, create copies of themselves, and adapt to novel challenges they encounter in the wild.

We refer to this cluster of capabilities as "autonomous replication and adaptation" or ARA.
We believe that systems capable of ARA could have wide-reaching and hard-to-anticipate consequences, and that measuring and forecasting ARA may be useful for informing measures around security, monitoring, and alignment.
Additionally, once a system is capable of ARA, placing bounds on a system's capabilities may become significantly more difficult.

Evaluating Language-Model Agents on Realistic Autonomous Tasks · Around

Built on

“Measuring Coding Challenge Competence With APPS”
Dan Hendrycks et al · 2021
Earlier work this paper cites.
“Measuring Massive Multitask Language Understanding”
Dan Hendrycks et al · 2021
Earlier work this paper cites.
“Measuring Mathematical Problem Solving With the MATH Dataset”
Dan Hendrycks et al · 2021
Earlier work this paper cites.
“Are we learning yet? a meta review of evaluation failures across machine learning”
Thomas Liao, Rohan Taori, Inioluwa Raji and Ludwig Schmidt · 2021
Earlier work this paper cites.
“Holistic evaluation of language models”
Original
Percy Liang et al · 2022
Earlier work this paper cites.
“Chain of thought prompting elicits reasoning in large language models”
Original
Jason Wei et al · 2022
Earlier work this paper cites.
“Mind2Web: Towards a Generalist Agent for the Web” type: article, 2023
Original
Xiang Deng et al · 2023
Earlier work this paper cites.

Similar

“Repairing the cracked foundation: A survey of obstacles in evaluation practices for generated text”
Sebastian Gehrmann, Elizabeth Clark and Thibault Sellam · 2023
Cited alongside, same era.
“BabyAGI” original-date: 2023-04-03T00:40:27Z, 2023
Yohei Nakajima · 2023
Cited alongside, same era.
“ChatGPT plugins”, 2023
OpenAI · 2023
Cited alongside, same era.
“GPT-4 Technical Report”
Original
OpenAI · 2023
Cited alongside, same era.
“Auto-GPT: An Autonomous GPT-4 Experiment” original-date: 2023-03-16T09:21:07Z
Toran Richards · 2023
Cited alongside, same era.
“Are Emergent Abilities of Large Language Models a Mirage?” type: article, 2023
Original

Then

“Reflexion: Language Agents with Verbal Reinforcement Learning” type: article, 2023
Original
Noah Shinn et al · 2023
Closest in time.
“Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models” type: article, 2023
Original
Aarohi Srivastava et al · 2023
Closest in time.
“SuperAGI” original-date: 2023-05-13T08:55:24Z, 2023
SuperAGI · 2023
Closest in time.
“Voyager: An Open-Ended Embodied Agent with Large Language Models” type: article, 2023
Original
Guanzhi Wang et al · 2023
Closest in time.
“ReAct: Synergizing Reasoning and Acting in Language Models”
Shunyu Yao et al · 2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Rylan Schaeffer, Brando Miranda and Sanmi Koyejo · 2023

Cited alongside, same era.

“Model evaluation for extreme risks”

Toby Shevlane et al · 2023

Cited alongside, same era.

“WebArena: A Realistic Web Environment for Building Autonomous Agents”, 2023

Shuyan Zhou et al · 2023

Closest in time.