FSPO: Few-Shot Optimization of Synthetic Preferences Personalizes to Real Users

Understand

Effective personalization of LLMs is critical for a broad range of user-interfacing applications such as virtual assistants and content curation.

• Inspired by the strong in-context capabilities of LLMs, we propose few-shot preference optimization (FSPO), an algorithm for LLM personalization that reframes reward modeling as a meta-learning problem.
• Under FSPO, an LLM learns to quickly infer a personalized reward function for a user via a few labeled preferences.
• FSPO also utilizes user description rationalization (RAT) to encourage better reward modeling and instruction following, recovering performance with the oracle user description.