Mechanism Design for LLM Fine-tuning with Multiple Reward Models

Understand

Fine-tuning large language models (LLMs) to aggregate multiple preferences has attracted considerable research attention.

• With aggregation algorithms advancing, a potential economic scenario arises where fine-tuning services are provided to agents with different preferences.
• In this context, agents may benefit from strategically misreporting their preferences, but this could harm the aggregation performance.
• This paper addresses such incentive issues by framing it as a mechanism design problem: an LLM provider determines the fine-tuning objective (training rule) and the pricing scheme (payment rule) for agents.