AgentPRM: Process Reward Models for LLM Agents via Step-Wise Promise and Progress

Understand

Despite rapid development, large language models (LLMs) still encounter challenges in multi-turn decision-making tasks (i.e., agent tasks) like web shopping and browser navigation, which require making a sequence of intelligent decisions based on environmental feedback.

• Previous work for LLM agents typically relies on elaborate prompt engineering or fine-tuning with expert trajectories to improve performance.
• In this work, we take a different perspective: we explore constructing process reward models (PRMs) to evaluate each decision and guide the agent's decision-making process.
• Unlike LLM reasoning, where each step is scored based on correctness, actions in agent tasks do not have a clear-cut correctness.