TS-Align: A Teacher-Student Collaborative Framework for Scalable Iterative Finetuning of Large Language Models

Understand

Mainstream approaches to aligning large language models (LLMs) heavily rely on human preference data, particularly when models require periodic updates.

• The standard process for iterative alignment of LLMs involves collecting new human feedback for each update.
• However, the data collection process is costly and challenging to scale.
• To address this issue, we introduce the "TS-Align" framework, which fine-tunes a policy model using pairwise feedback data automatically mined from its outputs.