EPO: Hierarchical LLM Agents with Environment Preference Optimization

Understand

Long-horizon decision-making tasks present significant challenges for LLM-based agents due to the need for extensive planning over multiple steps.

• In this paper, we propose a hierarchical framework that decomposes complex tasks into manageable subgoals, utilizing separate LLMs for subgoal prediction and low-level action generation.
• To address the challenge of creating training signals for unannotated datasets, we develop a reward model that leverages multimodal environment feedback to automatically generate reward signals.
• We introduce Environment Preference Optimization (EPO), a novel method that generates preference signals from the environment's feedback and uses them to train LLM-based agents.