Supervised Fine-Tuning as Inverse Reinforcement Learning

Understand

The prevailing approach to aligning Large Language Models (LLMs) typically relies on human or AI feedback and assumes access to specific types of preference datasets.

• In our work, we question the efficacy of such datasets and explore various scenarios where alignment with expert demonstrations proves more realistic.
• We build a sequential decision-making framework to formulate the problem of aligning LLMs using demonstration datasets.
• Drawing insights from inverse reinforcement learning and imitation learning, we introduce various approaches for divergence minimization in the LLM alignment tasks.