Fetching the paper…

Reading the bibliography…

2019

Scene Graph based Image Retrieval -- A case study on the CLEVR Dataset

Ramnath, Sahana, Saha, Amrita, Chakrabarti, Soumen et al.

Original
PDF
Post on X

Understand

With the prolification of multimodal interaction in various domains, recently there has been much interest in text based image retrieval in the computer vision community.

However most of the state of the art techniques model this problem in a purely neural way, which makes it difficult to incorporate pragmatic strategies in searching a large scale catalog especially when the search requirements are insufficient and the model needs to resort to an interactive retrieval process through multiple iterations of question-answering.
Motivated by this, we propose a neural-symbolic approach for a one-shot retrieval of images from a large scale catalog, given the caption description.
To facilitate this, we represent the catalog and caption as scene-graphs and model the retrieval task as a learnable graph matching problem, trained end-to-end with a REINFORCE algorithm.

Scene Graph based Image Retrieval -- A case study on the CLEVR Dataset · Around

Built on

Policy gradient methods for reinforcement learning with function approximation
Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour · 2000
Earlier work this paper cites.
Glove: Global vectors for word representation
Jeffrey Pennington, Richard Socher, and Christopher Manning · 2014
Earlier work this paper cites.
Image retrieval using scene graphs
Justin Johnson, Ranjay Krishna, Michael Stark, Li-Jia Li, David Shamma, Michael Bernstein, and Li Fei-Fei · 2015
Earlier work this paper cites.

Similar

Spice: Semantic propositional image caption evaluation
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould · 2016
Cited alongside, same era.
Clevr: A diagnostic dataset for compositional language and elementary visual reasoning
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick · 2017
Cited alongside, same era.

Then

Clevr-dialog: A diagnostic dataset for multi-round reasoning in visual dialog
Original
Satwik Kottur, José MF Moura, Devi Parikh, Dhruv Batra, and Marcus Rohrbach · 2019
Closest in time.
Composing text and image for image retrieval-an empirical odyssey
Nam Vo, Lu Jiang, Chen Sun, Kevin Murphy, Li-Jia Li, Li Fei-Fei, and James Hays · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…