Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedback

Understand

Large language models (LLMs) are used to generate content for a wide range of tasks, and are set to reach a growing audience in coming years due to integration in product interfaces like ChatGPT or search engines like Bing.

• This intensifies the need to ensure that models are aligned with human preferences and do not produce unsafe, inaccurate or toxic outputs.
• While alignment techniques like reinforcement learning with human feedback (RLHF) and red-teaming can mitigate some safety concerns and improve model capabilities, it is unlikely that an aggregate fine-tuning process can adequately represent the full range of users' preferences and values.
• Different people may legitimately disagree on their preferences for language and conversational norms, as well as on values or ideologies which guide their communication.