2018

Interactive Visual Grounding of Referring Expressions for Human-Robot Interaction

Shridhar, Mohit, Hsu, David

Understand

This paper presents INGRESS, a robot system that follows human natural language instructions to pick and place everyday objects.

The core issue here is the grounding of referring expressions: infer objects and their relationships from input images and language expressions.
INGRESS allows for unconstrained object categories and unconstrained language expressions.
Further, it asks questions to disambiguate referring expressions interactively.

Built on

Grounding in communication
H. Clark et al · 1991
Earlier work this paper cites.
Understanding referring expressions in a person-machine spoken dialogue
C. Pateras, G. Dudek, and R. De Mori · 1995
Earlier work this paper cites.
Meteor: An automatic metric for mt evaluation with improved correlation with human judgments
S. Banerjee and A. Lavie · 2005
Earlier work this paper cites.
A game-theoretic approach to generating spatial descriptions
D. Golland, P. Liang, and D. Klein · 2010
Earlier work this paper cites.
Toward understanding natural language directions
T. Kollar, S. Tellex, D. Roy, and N. Roy · 2010
Earlier work this paper cites.
Grounding spatial language for video search
S. Tellex, T. Kollar, G. Shaw, N. Roy, and D. Roy · 2010
Earlier work this paper cites.
A Joint Model of Language and Perception for Grounded Attribute Learning
C. Matuszek, N. FitzGerald, L. Zettlemoyer, L. Bo, and D. Fox · 2012
Earlier work this paper cites.
Semantic object maps for robotic housework-representation, acquisition and use
D. Pangercic, B. Pitzer, M. Tenorth, and M. Beetz · 2012
Earlier work this paper cites.
The Oxford handbook of compositionality
M. Werning, W. Hinzen, and E. Machery · 2012
Earlier work this paper cites.
Learning distributions over logical forms for referring expression generation
N. FitzGerald, Y. Artzi, and L. S. Zettlemoyer · 2013
Earlier work this paper cites.
Grounding spatial relations for human-robot interaction
S. Guadarrama, L. Riano, D. Golland, D. Go, Y. Jia, D. Klein, P. Abbeel, T. Darrell, et al · 2013
Earlier work this paper cites.

Similar

Multiscale combinatorial grouping
P. Arbeláez, J. Pont-Tuset, J. T. Barron, F. Marques, and J. Malik · 2014
Cited alongside, same era.
Referitgame: Referring to objects in photographs of natural scenes
S. Kazemzadeh, V. Ordonez, M. Matten, and T. L. Berg · 2014
Cited alongside, same era.
Cognitive psychology
U. Neisser · 2014
Cited alongside, same era.
Communication accommodation theory
C. Gallois and H. Giles · 2015
Cited alongside, same era.
Natural language communication with robots
Y. Bisk, D. Yuret, and D. Marcu · 2016
Cited alongside, same era.
Lessons from the amazon picking challenge: Four aspects of building robotic systems
C. Eppner, S. Höfer, R. Jonschkowski, R. Martin, A. Sieverling, V. Wall, and O. Brock · 2016
Cited alongside, same era.
Markerless perspective taking for humanoid robots in unconstrained environments
T. Fischer and Y. Demiris · 2016
Cited alongside, same era.
Natural language object retrieval
R. Hu, H. Xu, M. Rohrbach, J. Feng, K. Saenko, and T. Darrell · 2016
Cited alongside, same era.
Natural spatial description generation for human-robot interaction in indoor environments
Z. Huo and M. Skubic · 2016
Cited alongside, same era.
Densecap: Fully convolutional localization networks for dense captioning
J. Johnson, A. Karpathy, and L. Fei-Fei · 2016
Cited alongside, same era.
Visual genome: Connecting language and vision using crowdsourced dense image annotations
R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L.-J. Li, D. A. Shamma, M. Bernstein, and L. Fei-Fei · 2016
Cited alongside, same era.
Act to see and see to act: Pomdp planning for objects search in clutter
J. K. Li, D. Hsu, and W. S. Lee
Cited in the paper.

Then

Generation and comprehension of unambiguous object descriptions
J. Mao, J. Huang, A. Toshev, O. Camburu, A. L. Yuille, and K. Murphy · 2016
Later among the works it cites.
Modeling context between objects for referring expression understanding
V. K. Nagaraja, V. I. Morariu, and L. S. Davis · 2016
Later among the works it cites.
Robot reading human gaze: Why eye tracking is better than head tracking for human-robot collaboration
O. Palinko, F. Rea, G. Sandini, and A. Sciutti · 2016
Later among the works it cites.
Efficient grounding of abstract spatial concepts for natural language interaction with robot manipulators
R. Paul, J. Arkin, N. Roy, and T. Howard · 2016
Later among the works it cites.
Modeling relationships in referential expressions with compositional modular networks
R. Hu, M. Rohrbach, J. Andreas, T. Darrell, and K. Saenko · 2017
Later among the works it cites.
Clevr: A diagnostic dataset for compositional language and elementary visual reasoning
J. Johnson, B. Hariharan, L. van der Maaten, L. Fei-Fei, C. L. Zitnick, and R. Girshick · 2017
Later among the works it cites.
Dex-net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics
J. Mahler, J. Liang, S. Niyaz, M. Laskey, R. Doan, X. Liu, J. Aparicio, and K. Goldberg · 2017
Later among the works it cites.
A simple neural network module for relational reasoning
A. Santoro, D. Raposo, D. G. Barrett, M. Malinowski, R. Pascanu, P. Battaglia, and T. Lillicrap · 2017
Later among the works it cites.
A corpus of natural language for visual reasoning
A. Suhr, M. Lewis, J. Yeh, and Y. Artzi · 2017
Later among the works it cites.
A joint speaker-listener-reinforcer model for referring expressions
L. Yu, H. Tan, M. Bansal, and T. L. Berg · 2017
Later among the works it cites.
Interactively picking real-world objects with unconstrained spoken language instructions
J. Hatori, Y. Kikuchi, S. Kobayashi, K. Takahashi, Y. Tsuboi, Y. Unno, W. Ko, and J. Tan · 2018
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…