Latent Adversarial Regularization for Offline Preference Optimization

Understand

Learning from human feedback typically relies on preference optimization that constrains policy updates through token-level regularization.

• However, preference optimization for language models is particularly challenging because token-space similarity does not imply semantic or behavioral similarity.
• To address this challenge, we leverage latent-space regularization for language model preference optimization.
• We introduce GANPO, which achieves latent-space regularization by penalizing divergence between the internal representations of a policy model and a reference model.