Improving Context-Aware Preference Modeling for Language Models

Understand

While finetuning language models from pairwise preferences has proven remarkably effective, the underspecified nature of natural language presents critical challenges.

• Direct preference feedback is uninterpretable, difficult to provide where multidimensional criteria may apply, and often inconsistent, either because it is based on incomplete instructions or provided by diverse principals.
• To address these challenges, we consider the two-step preference modeling procedure that first resolves the under-specification by selecting a context, and then evaluates preference with respect to the chosen context.
• We decompose reward modeling error according to these two steps, which suggests that supervising context in addition to context-specific preference may be a viable approach to aligning models with diverse human preferences.