Visually-Prompted Language Model for Fine-Grained Scene Graph Generation in an Open World

Understand

Scene Graph Generation (SGG) aims to extract <subject, predicate, object> relationships in images for vision understanding.

• Although recent works have made steady progress on SGG, they still suffer long-tail distribution issues that tail-predicates are more costly to train and hard to distinguish due to a small amount of annotated data compared to frequent predicates.
• Existing re-balancing strategies try to handle it via prior rules but are still confined to pre-defined conditions, which are not scalable for various models and datasets.
• In this paper, we propose a Cross-modal prediCate boosting (CaCao) framework, where a visually-prompted language model is learned to generate diverse fine-grained predicates in a low-resource way.