End-to-end Dense Video Captioning as Sequence Generation

Understand

Dense video captioning aims to identify the events of interest in an input video, and generate descriptive captions for each event.

• Previous approaches usually follow a two-stage generative process, which first proposes a segment for each event, then renders a caption for each identified segment.
• Recent advances in large-scale sequence generation pretraining have seen great success in unifying task formulation for a great variety of tasks, but so far, more complex tasks such as dense video captioning are not able to fully utilize this powerful paradigm.
• In this work, we show how to model the two subtasks of dense video captioning jointly as one sequence generation task, and simultaneously predict the events and the corresponding descriptions.