Web-Shepherd: Advancing PRMs for Reinforcing Web Agents

Understand

Web navigation is a unique domain that can automate many repetitive real-life tasks and is challenging as it requires long-horizon sequential decision making beyond typical multimodal large language model (MLLM) tasks.

• Yet, specialized reward models for web navigation that can be utilized during both training and test-time have been absent until now.
• Despite the importance of speed and cost-effectiveness, prior works have utilized MLLMs as reward models, which poses significant constraints for real-world deployment.
• To address this, in this work, we propose the first process reward model (PRM) called Web-Shepherd which could assess web navigation trajectories in a step-level.