Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-control

Understand

This paper proposes a general method for improving the structure and quality of sequences generated by a recurrent neural network (RNN), while maintaining information originally learned from data, as well as sample diversity.

• An RNN is first pre-trained on data using maximum likelihood estimation (MLE), and the probability distribution over the next token in the sequence learned by this model is treated as a prior policy.
• Another RNN is then trained using reinforcement learning (RL) to generate higher-quality outputs that account for domain-specific incentives while retaining proximity to the prior policy of the MLE RNN.
• To formalize this objective, we derive novel off-policy RL methods for RNNs from KL-control.