Learning Personalized Alignment for Evaluating Open-ended Text Generation

Understand

Recent research has increasingly focused on evaluating large language models' (LLMs) alignment with diverse human values and preferences, particularly for open-ended tasks like story generation.

• Traditional evaluation metrics rely heavily on lexical similarity with human-written references, often showing poor correlation with human judgments and failing to account for alignment with the diversity of human preferences.
• To address these challenges, we introduce PerSE, an interpretable evaluation framework designed to assess alignment with specific human preferences.
• It is tuned to infer specific preferences from an in-context personal profile and evaluate the alignment between the generated content and personal preferences.