Using Syntax to Ground Referring Expressions in Natural Images

Understand

We introduce GroundNet, a neural network for referring expression recognition -- the task of localizing (or grounding) in an image the object referred to by a natural language expression.

• Our approach to this task is the first to rely on a syntactic analysis of the input referring expression in order to inform the structure of the computation graph.
• Given a parse tree for an input expression, we explicitly map the syntactic constituents and relationships present in the tree to a composed graph of neural modules that defines our architecture for performing localization.
• This syntax-based approach aids localization of \textit{both} the target object and auxiliary supporting objects mentioned in the expression.