2023

NOPA: Neurally-guided Online Probabilistic Assistance for Building Socially Intelligent Home Assistants

Puig, Xavier, Shu, Tianmin, Tenenbaum, Joshua B. et al.

Understand

In this work, we study how to build socially intelligent robots to assist people in their homes.

In particular, we focus on assistance with online goal inference, where robots must simultaneously infer humans' goals and how to help them achieve those goals.
Prior assistance methods either lack the adaptivity to adjust helping strategies (i.e., when and how to help) in response to uncertainty about goals or the scalability to conduct fast inference in a large goal space.
Our NOPA (Neurally-guided Online Probabilistic Assistance) method addresses both of these challenges.

Built on

S. Carberry, “Techniques for plan recognition,” User modeling and user-adapted interaction , vol. 11, no. 1, pp. 31–48, 2001
2001
Earlier work this paper cites.
M. A. Goodrich and A. C. Schultz, “Human-robot interaction: a survey,” Foundations and trends in human-computer interaction , vol. 1, no. 3, pp. 203–275, 2007
2007
Earlier work this paper cites.
M. Ramírez and H. Geffner, “Plan recognition as planning,” in Twenty-First international joint conference on artificial intelligence , 2009
2009
Earlier work this paper cites.
T. Ullman, C. Baker, O. Macindoe, O. Evans, N. Goodman, and J. Tenenbaum, “Help or hinder: Bayesian models of social goal inference,” Advances in neural information processing systems , vol. 22, 2009
2009
Earlier work this paper cites.
S. C. Ong, S. W. Png, D. Hsu, and W. S. Lee, “Planning under uncertainty for robotic tasks with mixed observability,” The International Journal of Robotics Research , vol. 29, no. 8, pp. 1053–1068, 2010
2010
Earlier work this paper cites.
A. Fern, S. Natarajan, K. Judah, and P. Tadepalli, “A decision-theoretic model of assistance,” Journal of Artificial Intelligence Research , vol. 50, pp. 71–104, 2014
2014
Earlier work this paper cites.
D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980 , 2014
Original
2014
Earlier work this paper cites.
S. Nikolaidis, R. Ramakrishnan, K. Gu, and J. Shah, “Efficient model learning from joint-action demonstrations for human-robot collaborative tasks,” in 2015 10th ACM/IEEE International Conference on Human-Robot Interaction (HRI) . IEEE, 2015, pp. 189–196
2015
Earlier work this paper cites.
J. Johnson, R. Krishna, M. Stark, L.-J. Li, D. Shamma, M. Bernstein, and L. Fei-Fei, “Image retrieval using scene graphs,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2015, pp. 3668–3678
2015
Earlier work this paper cites.
S. Sohrabi, A. V. Riabov, and O. Udrea, “Plan recognition as planning revisited.” in IJCAI , 2016, pp. 3258–3264
2016
Earlier work this paper cites.
L. Rozo, S. Calinon, D. G. Caldwell, P. Jimenez, and C. Torras, “Learning physical collaborative robot behaviors from human demonstrations,” IEEE Transactions on Robotics , vol. 32, no. 3, pp. 513–527, 2016
2016
Earlier work this paper cites.
C. Liu, J. B. Hamrick, J. F. Fisac, A. D. Dragan, J. K. Hedrick, S. S. Sastry, and T. L. Griffiths, “Goal inference improves objective and perceived performance in human-robot collaboration,” in 15th International Conference on Autonomous Agents and Multiagent Systems (AAMAS) , 2016
2016
Earlier work this paper cites.
D. Hadfield-Menell, S. J. Russell, P. Abbeel, and A. Dragan, “Cooperative inverse reinforcement learning,” Advances in neural information processing systems , vol. 29, 2016
2016
Earlier work this paper cites.
C. L. Baker, J. Jara-Ettinger, R. Saxe, and J. B. Tenenbaum, “Rational quantitative attribution of beliefs, desires and percepts in human mentalizing,” Nature Human Behaviour , vol. 1, no. 4, pp. 1–10, 2017
2017
Earlier work this paper cites.

Similar

P. A. Lasota, T. Fong, J. A. Shah, et al. , A survey of methods for safe human-robot interaction . Now Publishers Delft, The Netherlands, 2017, vol. 104
2017
Cited alongside, same era.
N. Rabinowitz, F. Perbet, F. Song, C. Zhang, S. A. Eslami, and M. Botvinick, “Machine theory of mind,” in International conference on machine learning . PMLR, 2018, pp. 4218–4227
2018
Cited alongside, same era.
S. V. Albrecht and P. Stone, “Autonomous agents modelling other agents: A comprehensive survey and open problems,” Artificial Intelligence , vol. 258, pp. 66–95, 2018
2018
Cited alongside, same era.
S. Javdani, H. Admoni, S. Pellegrinelli, S. S. Srinivasa, and J. A. Bagnell, “Shared autonomy via hindsight optimization for teleoperation and teaming,” The International Journal of Robotics Research , vol. 37, no. 7, pp. 717–742, 2018
2018
Cited alongside, same era.
M. Carroll, R. Shah, M. K. Ho, T. Griffiths, S. Seshia, P. Abbeel, and A. Dragan, “On the utility of learning about humans for human-AI coordination,” Advances in neural information processing systems , vol. 32, 2019
2019
Cited alongside, same era.
M. Jaderberg, W. M. Czarnecki, I. Dunning, L. Marris, G. Lever, A. G. Castaneda, C. Beattie, N. C. Rabinowitz, A. S. Morcos, A. Ruderman, et al. , “Human-level performance in 3d multiplayer games with population-based reinforcement learning,” Science , vol. 364, no. 6443, pp. 859–865, 2019
2019
Cited alongside, same era.
Y.-H. Liao, X. Puig, M. Boben, A. Torralba, and S. Fidler, “Synthesizing environment-aware activities via activity sketches,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2019, pp. 6291–6299
2019
Cited alongside, same era.
Z. Cao, H. Gao, K. Mangalam, Q.-Z. Cai, M. Vo, and J. Malik, “Long-term human motion prediction with scene context,” in European Conference on Computer Vision . Springer, 2020, pp. 387–404
2020
Cited alongside, same era.
B. Liu, E. Adeli, Z. Cao, K.-H. Lee, A. Shenoi, A. Gaidon, and J. C. Niebles, “Spatiotemporal relationship reasoning for pedestrian intent prediction,” IEEE Robotics and Automation Letters , vol. 5, no. 2, pp. 3485–3492, 2020
2020
Cited alongside, same era.
Z. Nan, T. Shu, R. Gong, S. Wang, P. Wei, S.-C. Zhu, and N. Zheng, “Learning to infer human attention in daily activities,” Pattern Recognition , vol. 103, p. 107314, 2020
2020
Cited alongside, same era.
P. Dendorfer, A. Osep, and L. Leal-Taixé, “Goal-GAN: Multimodal trajectory prediction based on goal position estimation,” in Proceedings of the Asian Conference on Computer Vision , 2020
2020
Cited alongside, same era.
H. Zhao, J. Gao, T. Lan, C. Sun, B. Sapp, B. Varadarajan, Y. Shen, Y. Shen, Y. Chai, C. Schmid, et al. , “TNT: Target-driven trajectory prediction,” arXiv preprint arXiv:2008.08294 , 2020
Original
2020
Cited alongside, same era.

Then

H. Hu, A. Lerer, A. Peysakhovich, and J. Foerster, ““other-play” for zero-shot coordination,” in International Conference on Machine Learning . PMLR, 2020, pp. 4399–4410
2020
Later among the works it cites.
A. Shah, S. Li, and J. Shah, “Planning with uncertain specifications (PUnS),” IEEE Robotics and Automation Letters , vol. 5, no. 2, pp. 3414–3421, 2020
2020
Later among the works it cites.
Y. Du, S. Tiomkin, E. Kiciman, D. Polani, P. Abbeel, and A. Dragan, “AVE: Assistance via empowerment,” Advances in Neural Information Processing Systems , vol. 33, pp. 4560–4571, 2020
2020
Later among the works it cites.
F. Yan, D. Wang, and H. He, “Robotic understanding of spatial relationships using neural-logic learning,” in 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) . IEEE, 2020, pp. 8358–8365
2020
Later among the works it cites.
S. Nguyen, O. S. Oguz, V. N. Hartmann, and M. Toussaint, “Self-supervised learning of scene-graph representations for robotic sequential manipulation planning.” in CoRL , 2020, pp. 2104–2119
2020
Later among the works it cites.
X. Puig, T. Shu, S. Li, Z. Wang, Y.-H. Liao, J. B. Tenenbaum, S. Fidler, and A. Torralba, “Watch-And-Help: A challenge for social perception and human-AI collaboration,” in International Conference on Learning Representations , 2021
2021
Later among the works it cites.
Y. Yao, E. Atkins, M. Johnson-Roberson, R. Vasudevan, and X. Du, “BITRAP: Bi-directional pedestrian trajectory prediction with multi-modal goal estimation,” IEEE Robotics and Automation Letters , vol. 6, no. 2, pp. 1463–1470, 2021
2021
Later among the works it cites.
H. Tran, V. Le, and T. Tran, “Goal-driven long-term trajectory prediction,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , 2021, pp. 796–805
2021
Later among the works it cites.
K. Mangalam, Y. An, H. Girase, and J. Malik, “From goals, waypoints & paths to long term human trajectory forecasting,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 15 233–15 242
2021
Later among the works it cites.
A. Netanyahu, T. Shu, B. Katz, A. Barbu, and J. B. Tenenbaum, “PHASE: Physically-grounded abstract social events for machine social perception,” in 35th AAAI Conference on Artificial Intelligence (AAAI) , 2021
2021
Later among the works it cites.
S. A. Wu, R. E. Wang, J. A. Evans, J. B. Tenenbaum, D. C. Parkes, and M. Kleiman-Weiner, “Too many cooks: Bayesian inference for coordinating multi-agent collaboration,” Topics in Cognitive Science , vol. 13, no. 2, pp. 414–432, 2021
2021
Later among the works it cites.
Y. Zhu, J. Tremblay, S. Birchfield, and Y. Zhu, “Hierarchical planning for long-horizon manipulation with geometric and symbolic scene graphs,” in 2021 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2021, pp. 6541–6548
2021
Later among the works it cites.
R. Tejwani, Y.-L. Kuo, T. Shu, B. Katz, and A. Barbu, “Social interactions as recursive mdps,” in Conference on Robot Learning . PMLR, 2022, pp. 949–958
2022
Later among the works it cites.
S. Srivastava, C. Li, M. Lingelbach, R. Martín-Martín, F. Xia, K. E. Vainio, Z. Lian, C. Gokmen, S. Buch, K. Liu, et al. , “Behavior: Benchmark for everyday household activities in virtual, interactive, and ecological environments,” in Conference on Robot Learning . PMLR, 2022, pp. 477–490
2022
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…