Automatic Annotation of Structured Facts in Images

Understand

Motivated by the application of fact-level image understanding, we present an automatic method for data collection of structured visual facts from images with captions.

• Example structured facts include attributed objects (e.g., <flower, red>), actions (e.g., <baby, smile>), interactions (e.g., <man, walking, dog>), and positional information (e.g., <vase, on, table>).
• The collected annotations are in the form of fact-image pairs (e.g.,<man, walking, dog> and an image region containing this fact).
• With a language approach, the proposed method is able to collect hundreds of thousands of visual fact annotations with accuracy of 83% according to human judgment.