Consensus-based Sequence Training for Video Captioning

Understand

Captioning models are typically trained using the cross-entropy loss.

• However, their performance is evaluated on other metrics designed to better correlate with human assessments.
• Recently, it has been shown that reinforcement learning (RL) can directly optimize these metrics in tasks such as captioning.
• However, this is computationally costly and requires specifying a baseline reward at each step to make training converge.