Transferring Knowledge from Vision to Language: How to Achieve it and how to Measure it?

Understand

Large language models are known to suffer from the hallucination problem in that they are prone to output statements that are false or inconsistent, indicating a lack of knowledge.

• A proposed solution to this is to provide the model with additional data modalities that complements the knowledge obtained through text.
• We investigate the use of visual data to complement the knowledge of large language models by proposing a method for evaluating visual knowledge transfer to text for uni- or multimodal language models.
• The method is based on two steps, 1) a novel task querying for knowledge of memory colors, i.e.