Less Is More: Picking Informative Frames for Video Captioning

Understand

In video captioning task, the best practice has been achieved by attention-based models which associate salient visual components with sentences in the video.

• However, existing study follows a common procedure which includes a frame-level appearance modeling and motion modeling on equal interval frame sampling, which may bring about redundant visual information, sensitivity to content noise and unnecessary computation cost.
• We propose a plug-and-play PickNet to perform informative frame picking in video captioning.
• Based on a standard Encoder-Decoder framework, we develop a reinforcement-learning-based procedure to train the network sequentially, where the reward of each frame picking action is designed by maximizing visual diversity and minimizing textual discrepancy.