A Self-Supervised Reinforcement Learning Approach for Fine-Tuning Large Language Models Using Cross-Attention Signals

Understand

We propose a novel reinforcement learning framework for post training large language models that does not rely on human in the loop feedback.

• Instead, our approach uses cross attention signals within the model itself to derive a self supervised reward, thereby guiding iterative fine tuning of the model policy.
• By analyzing how the model attends to the input prompt during generation, we construct measures of prompt coverage, focus, and coherence.
• We then use these measures to rank or score candidate responses, providing a reward signal that encourages the model to produce well aligned, on topic text.