2021

AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting

Yuan, Ye, Weng, Xinshuo, Ou, Yanglan et al.

Understand

Predicting accurate future trajectories of multiple agents is essential for autonomous systems, but is challenging due to the complex agent interaction and the uncertainty in each agent's future behavior.

Forecasting multi-agent trajectories requires modeling two key dimensions: (1) time dimension, where we model the influence of past agent states over future states; (2) social dimension, where we model how the state of each agent affects others.
Most prior methods model these two dimensions separately, e.g., first using a temporal model to summarize features over time for each agent independently and then modeling the interaction of the summarized features with a social model.
This approach is suboptimal since independent feature encoding over either the time or social dimension can result in a loss of information.

Built on

Social force model for pedestrian dynamics
Dirk Helbing and Peter Molnar · 1995
Earlier work this paper cites.
Long short-term memory
Sepp Hochreiter and Jürgen Schmidhuber · 1997
Earlier work this paper cites.
Crowds by example
Alon Lerner, Yiorgos Chrysanthou, and Dani Lischinski · 2007
Earlier work this paper cites.
Gaussian process dynamical models for human motion
Jack M Wang, David J Fleet, and Aaron Hertzmann · 2007
Earlier work this paper cites.
You’ll never walk alone: Modeling social behavior for multi-target tracking
Stefano Pellegrini, Andreas Ess, Konrad Schindler, and Luc Van Gool · 2009
Earlier work this paper cites.
Auto-encoding variational bayes
Original
Diederik P Kingma and Max Welling · 2013
Earlier work this paper cites.
Neural machine translation by jointly learning to align and translate
Original
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio · 2014
Earlier work this paper cites.
Learning phrase representations using rnn encoder-decoder for statistical machine translation
Original
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio · 2014
Earlier work this paper cites.
Empirical evaluation of gated recurrent neural networks on sequence modeling
Original
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio · 2014
Earlier work this paper cites.
Generative adversarial networks
Original
Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio · 2014
Earlier work this paper cites.
Adam: A method for stochastic optimization
Original
Diederik P Kingma and Jimmy Ba · 2014
Earlier work this paper cites.
Recurrent network models for human dynamics
Katerina Fragkiadaki, Sergey Levine, Panna Felsen, and Jitendra Malik · 2015
Earlier work this paper cites.
Gated graph sequence neural networks
Original
Yujia Li, Daniel Tarlow, Marc Brockschmidt, and Richard Zemel · 2015
Earlier work this paper cites.
Effective approaches to attention-based neural machine translation
Original
Minh-Thang Luong, Hieu Pham, and Christopher D Manning · 2015
Earlier work this paper cites.
Eesen: End-to-end speech recognition using deep rnn models and wfst-based decoding
Yajie Miao, Mohammad Gowayyed, and Florian Metze · 2015
Earlier work this paper cites.
Variational inference with normalizing flows
Danilo Rezende and Shakir Mohamed · 2015
Earlier work this paper cites.
Show, attend and tell: Neural image caption generation with visual attention
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio · 2015
Earlier work this paper cites.
Social lstm: Human trajectory prediction in crowded spaces
Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, and Silvio Savarese · 2016
Earlier work this paper cites.
Semi-supervised classification with graph convolutional networks
Original
Thomas N Kipf and Max Welling · 2016
Earlier work this paper cites.
Analysis of recurrent neural networks for probabilistic modeling of driver behavior
Jeremy Morton, Tim A Wheeler, and Mykel J Kochenderfer · 2016
Earlier work this paper cites.
Desire: Distant future prediction in dynamic scenes with interacting agents
Namhoon Lee, Wongun Choi, Paul Vernaza, Christopher B Choy, Philip HS Torr, and Manmohan Chandraker · 2017
Earlier work this paper cites.

Similar

Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin · 2017
Cited alongside, same era.
Bert: Pre-training of deep bidirectional transformers for language understanding
Original
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova · 2018
Cited alongside, same era.
Social gan: Socially acceptable trajectories with generative adversarial networks
Agrim Gupta, Justin Johnson, Li Fei-Fei, Silvio Savarese, and Alexandre Alahi · 2018
Cited alongside, same era.
Neural relational inference for interacting systems
Thomas Kipf, Ethan Fetaya, Kuan-Chieh Wang, Max Welling, and Richard Zemel · 2018
Cited alongside, same era.
R2p2: A reparameterized pushforward policy for diverse, precise generative path forecasting
Nicholas Rhinehart, Kris M Kitani, and Paul Vernaza · 2018
Cited alongside, same era.
Social attention: Modeling attention in human crowds
Anirudh Vemula, Katharina Muelling, and Jean Oh · 2018
Cited alongside, same era.
The microsoft 2017 conversational speech recognition system
Wayne Xiong, Lingfeng Wu, Fil Alleva, Jasha Droppo, Xuedong Huang, and Andreas Stolcke · 2018
Cited alongside, same era.
3d ego-pose estimation via imitation learning
Ye Yuan and Kris Kitani · 2018
Cited alongside, same era.
Multimodal trajectory predictions for autonomous driving using deep convolutional networks
Henggang Cui, Vladan Radosavljevic, Fang-Chieh Chou, Tsung-Han Lin, Thi Nguyen, Tzu-Kuo Huang, Jeff Schneider, and Nemanja Djuric · 2019
Cited alongside, same era.
Stgat: Modeling spatial-temporal interactions for human trajectory prediction
Yingfan Huang, Huikun Bi, Zhaoxin Li, Tianlu Mao, and Zhaoqi Wang · 2019
Cited alongside, same era.
The trajectron: Probabilistic multi-agent trajectory modeling with dynamic spatiotemporal graphs
Boris Ivanovic and Marco Pavone · 2019
Cited alongside, same era.
Social-bigat: multimodal trajectory forecasting using bicycle-gan and graph attention networks
Vineet Kosaraju, Amir Sadeghian, Roberto Martín-Martín, Ian D Reid, Hamid Rezatofighi, and Silvio Savarese · 2019
Cited alongside, same era.

Then

Multi-agent tensor fusion for contextual trajectory prediction
Tianyang Zhao, Yifei Xu, Mathew Monfort, Wongun Choi, Chris Baker, Yibiao Zhao, Yizhou Wang, and Ying Nian Wu · 2019
Later among the works it cites.
nuscenes: A multimodal dataset for autonomous driving
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom · 2020
Later among the works it cites.
End-to-end object detection with transformers
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko · 2020
Later among the works it cites.
Multipath: Multiple probabilistic anchor trajectory hypotheses for behavior prediction
Yuning Chai, Benjamin Sapp, Mayank Bansal, and Dragomir Anguelov · 2020
Later among the works it cites.
An image is worth 16x16 words: Transformers for image recognition at scale
Original
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al · 2020
Later among the works it cites.
Transformer networks for trajectory forecasting
Francesco Giuliari, Irtiza Hasan, Marco Cristani, and Fabio Galasso · 2020
Later among the works it cites.
Generative hybrid representations for activity forecasting with no-regret learning
Jiaqi Guan, Ye Yuan, Kris M Kitani, and Nicholas Rhinehart · 2020
Later among the works it cites.
Vibe: Video inference for human body pose and shape estimation
Muhammed Kocabas, Nikos Athanasiou, and Michael J Black · 2020
Later among the works it cites.
Evolvegraph: Multi-agent trajectory prediction with dynamic relational reasoning
Jiachen Li, Fan Yang, Masayoshi Tomizuka, and Chiho Choi · 2020
Later among the works it cites.
End-to-end contextual perception and prediction with interaction transformer
Original
Lingyun Luke Li, Bin Yang, Ming Liang, Wenyuan Zeng, Mengye Ren, Sean Segal, and Raquel Urtasun · 2020
Later among the works it cites.
Diverse sampling for normalizing flow based trajectory forecasting
Original
Yecheng Jason Ma, Jeevana Priya Inala, Dinesh Jayaraman, and Osbert Bastani · 2020
Later among the works it cites.
It is not the journey but the destination: Endpoint conditioned trajectory prediction
Karttikeya Mangalam, Harshayu Girase, Shreyas Agarwal, Kuan-Hui Lee, Ehsan Adeli, Jitendra Malik, and Adrien Gaidon · 2020
Later among the works it cites.
Covernet: Multimodal behavior prediction using trajectory sets
Tung Phan-Minh, Elena Corina Grigore, Freddy A Boulton, Oscar Beijbom, and Eric M Wolff · 2020
Later among the works it cites.
Human motion trajectory prediction: A survey
Andrey Rudenko, Luigi Palmieri, Michael Herman, Kris M Kitani, Dariu M Gavrila, and Kai O Arras · 2020
Later among the works it cites.
Trajectron++: Dynamically-feasible trajectory forecasting with heterogeneous data
Original
Tim Salzmann, Boris Ivanovic, Punarjay Chakravarty, and Marco Pavone · 2020
Later among the works it cites.
End-to-end video instance segmentation with transformers
Original
Yuqing Wang, Zhaoliang Xu, Xinlong Wang, Chunhua Shen, Baoshan Cheng, Hao Shen, and Huaxia Xia · 2020
Later among the works it cites.
Joint 3d tracking and forecasting with graph neural network and diversity sampling
Original
Xinshuo Weng, Ye Yuan, and Kris Kitani · 2020
Later among the works it cites.
Spatio-temporal graph transformer networks for pedestrian trajectory prediction
Cunjun Yu, Xiao Ma, Jiawei Ren, Haiyu Zhao, and Shuai Yi · 2020
Later among the works it cites.
Dlow: Diversifying latent flows for diverse human motion prediction
Ye Yuan and Kris Kitani · 2020
Later among the works it cites.
Residual force control for agile human behavior imitation and extended motion synthesis
Ye Yuan and Kris Kitani · 2020
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…