Zero-Shot Video Captioning with Evolving Pseudo-Tokens

Understand

We introduce a zero-shot video captioning method that employs two frozen networks: the GPT-2 language model and the CLIP image-text matching model.

• The matching score is used to steer the language model toward generating a sentence that has a high average matching score to a subset of the video frames.
• Unlike zero-shot image captioning methods, our work considers the entire sentence at once.
• This is achieved by optimizing, during the generation process, part of the prompt from scratch, by modifying the representation of all other tokens in the prompt, and by repeating the process iteratively, gradually improving the specificity and comprehensiveness of the generated sentence.