Preference VLM: Leveraging VLMs for Scalable Preference-Based Reinforcement Learning

Understand

Preference-based reinforcement learning (RL) offers a promising approach for aligning policies with human intent but is often constrained by the high cost of human feedback.

• In this work, we introduce PrefVLM, a framework that integrates Vision-Language Models (VLMs) with selective human feedback to significantly reduce annotation requirements while maintaining performance.
• Our method leverages VLMs to generate initial preference labels, which are then filtered to identify uncertain cases for targeted human annotation.
• Additionally, we adapt VLMs using a self-supervised inverse dynamics loss to improve alignment with evolving policies.