StoryTeller: Improving Long Video Description through Global Audio-Visual Character Identification

Understand

Existing large vision-language models (LVLMs) are largely limited to processing short, seconds-long videos and struggle with generating coherent descriptions for extended video spanning minutes or more.

• Long video description introduces new challenges, such as consistent character identification and plot-level descriptions incorporating both visual and audio information.
• To address these, we figure out audio-visual character identification, matching character names to each dialogue, as a key factor.
• We propose StoryTeller, a system for generating dense descriptions of long videos, incorporating both low-level visual concepts and high-level plot information.