RelTR: Relation Transformer for Scene Graph Generation

Understand

Different objects in the same scene are more or less related to each other, but only a limited number of these relationships are noteworthy.

• Inspired by DETR, which excels in object detection, we view scene graph generation as a set prediction problem and propose an end-to-end scene graph generation model RelTR which has an encoder-decoder architecture.
• The encoder reasons about the visual feature context while the decoder infers a fixed-size set of triplets subject-predicate-object using different types of attention mechanisms with coupled subject and object queries.
• We design a set prediction loss performing the matching between the ground truth and predicted triplets for the end-to-end training.