Zero-shot Visual Question Answering with Language Model Feedback

Understand

In this paper, we propose a novel language model guided captioning approach, LAMOC, for knowledge-based visual question answering (VQA).

• Our approach employs the generated captions by a captioning model as the context of an answer prediction model, which is a Pre-trained Language model (PLM).
• As the major contribution, we leverage the guidance and feedback of the prediction model to improve the capability of the captioning model.
• In this way, the captioning model can become aware of the task goal and information need from the PLM.