2019

Look, Listen, and Act: Towards Audio-Visual Embodied Navigation

Gan, Chuang, Zhang, Yiwei, Wu, Jiajun et al.

Understand

A crucial ability of mobile intelligent agents is to integrate the evidence from multiple sensory inputs in an environment and to make a sequence of actions to reach their goals.

In this paper, we attempt to approach the problem of Audio-Visual Embodied Navigation, the task of planning the shortest path from a random starting location in a scene to the sound source in an indoor environment, given only raw egocentric visual and audio sensory data.
To accomplish this task, the agent is required to learn from various modalities, i.e.
relating the audio signal to the visual environment.

Built on

H. McGurk and J. MacDonald, “Hearing lips and seeing voices,” Nat. , vol. 264, no. 5588, p. 746, 1976
1976
Earlier work this paper cites.
H. P. Moravec, “Sensor fusion in certainty grids for mobile robots,” in Sensor devices and systems for robotics . Springer, 1989, pp. 253–276
1989
Earlier work this paper cites.
J. R. Hershey and J. R. Movellan, “Audio vision: Using audio-visual synchrony to locate sounds,” in NeurIPS , 2000
2000
Earlier work this paper cites.
J. W. Fisher III, T. Darrell, W. T. Freeman, and P. A. Viola, “Learning joint statistical models for audio-visual fusion and segregation,” in NeurIPS , 2001
2001
Earlier work this paper cites.
H. L. Van Trees, Optimum array processing: Part IV of detection, estimation, and modulation theory . John Wiley & Sons, 2004
2004
Earlier work this paper cites.
S. Thrun, W. Burgard, and D. Fox, Probabilistic robotics . MIT press, 2005
2005
Earlier work this paper cites.
E. Kidron, Y. Y. Schechner, and M. Elad, “Pixels that sound,” in CVPR , 2005
2005
Earlier work this paper cites.
K. P. Körding, U. Beierholm, W. J. Ma, S. Quartz, J. B. Tenenbaum, and L. Shams, “Causal inference in multisensory perception,” PLOS ONE , vol. 2, no. 9, p. e943, 2007
2007
Earlier work this paper cites.
Z. Barzelay and Y. Y. Schechner, “Harmony in motion,” in CVPR , 2007
2007
Earlier work this paper cites.
H. Izadinia, I. Saleemi, and M. Shah, “Multimodal analysis for identification and segmentation of moving-sounding objects,” IEEE TMM , vol. 15, no. 2, pp. 378–390, 2013
2013
Earlier work this paper cites.
A. Zunino, M. Crocco, S. Martelli, A. Trucco, A. Del Bue, and V. Murino, “Seeing the sound: A new multimodal imaging device for computer vision,” in CVPR Workshop , 2015
2015
Earlier work this paper cites.
H. Mei, M. Bansal, and M. R. Walter, “Listen, attend, and walk: Neural mapping of navigational instructions to action sequences.” in AAAI , 2016
2016
Earlier work this paper cites.
A. Miller, A. Fisch, J. Dodge, A.-H. Karimi, A. Bordes, and J. Weston, “Key-value memory networks for directly reading documents,” in EMNLP , 2016
2016
Earlier work this paper cites.
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in CVPR , 2016
2016
Earlier work this paper cites.
V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu, “Asynchronous methods for deep reinforcement learning,” in ICML , 2016
2016
Earlier work this paper cites.

Similar

Y. Zhu, R. Mottaghi, E. Kolve, J. J. Lim, A. Gupta, L. Fei-Fei, and A. Farhadi, “Target-driven visual navigation in indoor scenes using deep reinforcement learning,” in ICRA , 2017
2017
Cited alongside, same era.
S. Gupta, J. Davidson, S. Levine, R. Sukthankar, and J. Malik, “Cognitive mapping and planning for visual navigation,” in CVPR , 2017
2017
Cited alongside, same era.
F. Sadeghi and S. Levine, “Cad2rl: Real single-image flight without a single real image,” in RSS , 2017
2017
Cited alongside, same era.
P. Mirowski, R. Pascanu, F. Viola, H. Soyer, A. J. Ballard, A. Banino, M. Denil, R. Goroshin, L. Sifre, K. Kavukcuoglu et al. , “Learning to navigate in complex environments,” in ICLR , 2017
2017
Cited alongside, same era.
S. Brahmbhatt and J. Hays, “Deepnav: Learning to navigate large cities,” in CVPR , 2017
2017
Cited alongside, same era.
Y. Wu and Y. Tian, “Training agent for first-person shooter game with actor-critic curriculum learning,” in ICLR , 2017
2017
Cited alongside, same era.
E. Kolve, R. Mottaghi, D. Gordon, Y. Zhu, A. Gupta, and A. Farhadi, “Ai2-thor: An interactive 3d environment for visual ai,” arXiv:1712.05474 , 2017
Original
2017
Cited alongside, same era.
N. Savinov, A. Dosovitskiy, and V. Koltun, “Semi-parametric topological memory for navigation,” in ICLR , 2018
2018
Cited alongside, same era.
D. Fried, R. Hu, V. Cirik, A. Rohrbach, J. Andreas, L.-P. Morency, T. Berg-Kirkpatrick, K. Saenko, D. Klein, and T. Darrell, “Speaker-follower models for vision-and-language navigation,” in NeurIPS , 2018
2018
Cited alongside, same era.
H. Yu, H. Zhang, and W. Xu, “Interactive grounded language acquisition and generalization in a 2d world,” in ICLR , 2018
2018
Cited alongside, same era.
P. Anderson, Q. Wu, D. Teney, J. Bruce, M. Johnson, N. Sünderhauf, I. Reid, S. Gould, and A. van den Hengel, “Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments,” in CVPR , 2018
2018
Cited alongside, same era.
R. Arandjelović and A. Zisserman, “Objects that sound,” in ECCV , 2018
2018
Cited alongside, same era.

Then

A. Owens and A. A. Efros, “Audio-visual scene analysis with self-supervised multisensory features,” in ECCV , 2018
2018
Later among the works it cites.
R. Gao, R. Feris, and K. Grauman, “Learning to separate object sounds by watching unlabeled video,” in ECCV , 2018
2018
Later among the works it cites.
H. Zhao, C. Gan, A. Rouditchenko, C. Vondrick, J. McDermott, and A. Torralba, “The sound of pixels,” in ECCV , 2018
2018
Later among the works it cites.
I. An, M. Son, D. Manocha, and S.-e. Yoon, “Reflection-aware sound source localization,” in ICRA , 2018
2018
Later among the works it cites.
P. Anderson, A. Chang, D. S. Chaplot, A. Dosovitskiy, S. Gupta, V. Koltun, J. Kosecka, J. Malik, R. Mottaghi, M. Savva et al. , “On evaluation of embodied navigation agents,” arXiv:1807.06757 , 2018
Original
2018
Later among the works it cites.
W. Yang, X. Wang, A. Farhadi, A. Gupta, and R. Mottaghi, “Visual semantic navigation using scene priors,” in ICLR , 2019
2019
Closest in time.
S. McLeod and J. Xiao, “Navigating dynamically unknown environments leveraging past experience,” in ICRA , 2019
2019
Closest in time.
K. Katyal, K. Popek, C. Paxton, P. Burlina, and G. D. Hager, “Uncertainty-aware occupancy map prediction using generative networks for robot navigation,” in ICRA , 2019
2019
Closest in time.
A. Mousavian, A. Toshev, M. Fiser, J. Kosecka, A. Wahid, and J. Davidson, “Visual representations for semantic target driven navigation,” in ICRA , 2019
2019
Closest in time.
C. Chen, U. Jain, C. Schissler, S. V. A. Gari, Z. Al-Halah, V. K. Ithapu, P. Robinson, and K. Grauman, “Audio-visual embodied navigation,” arXiv:1912.11474 , 2019
Original
2019
Closest in time.
M. Savva, A. Kadian, O. Maksymets, Y. Zhao, E. Wijmans, B. Jain, J. Straub, J. Liu, V. Koltun, J. Malik et al. , “Habitat: A platform for embodied AI research,” in ICCV , 2019
2019
Closest in time.
H. Zhao, C. Gan, W.-C. Ma, and A. Torralba, “The sound of motions,” in ICCV , 2019
2019
Closest in time.
A. Rouditchenko, H. Zhao, C. Gan, J. McDermott, and A. Torralba, “Self-supervised audio-visual co-segmentation,” in ICASSP , 2019
2019
Closest in time.
C. Gan, H. Zhao, P. Chen, D. Cox, and A. Torralba, “Self-supervised moving vehicle tracking with stereo sound,” in ICCV , 2019
2019
Closest in time.
M. Gorzel, A. Allen, I. Kelly, J. Kammerl, A. Gungormusler, H. Yeh, and F. Boland, “Efficient encoding and decoding of binaural sound with resonance audio,” in AES , 2019
2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…