Shatter and Gather: Learning Referring Image Segmentation with Text Supervision

Understand

Referring image segmentation, the task of segmenting any arbitrary entities described in free-form texts, opens up a variety of vision applications.

• However, manual labeling of training data for this task is prohibitively costly, leading to lack of labeled data for training.
• We address this issue by a weakly supervised learning approach using text descriptions of training images as the only source of supervision.
• To this end, we first present a new model that discovers semantic entities in input image and then combines such entities relevant to text query to predict the mask of the referent.