2018

Learning from Demonstration in the Wild

Behbahani, Feryal, Shiarlis, Kyriacos, Chen, Xi et al.

Understand

Learning from demonstration (LfD) is useful in settings where hand-coding behaviour or a reward function is impractical.

It has succeeded in a wide range of problems but typically relies on manually generated demonstrations or specially deployed sensors and has not generally been able to leverage the copious demonstrations available in the wild: those that capture behaviours that were occurring anyway using sensors that were already deployed for another purpose, e.g., traffic camera footage capturing demonstrations of natural behaviour of vehicles, cyclists, and pedestrians.
We propose Video to Behaviour (ViBe), a new approach to learn models of behaviour from unlabelled raw video data of a traffic scene collected from a single, monocular, initially uncalibrated camera with ordinary resolution.
Our approach calibrates the camera, detects relevant objects, tracks them through time, and uses the resulting trajectories to perform LfD, yielding models of naturalistic behaviour.

Built on

D. A. Pomerleau, “Alvinn: An autonomous land vehicle in a neural network,” in Advances in neural information processing systems , 1989, pp. 305–313
1989
Earlier work this paper cites.
M. Tan, “Multi-agent reinforcement learning: Independent vs. cooperative agents,” in Proceedings of the tenth international conference on machine learning , 1993, pp. 330–337
1993
Earlier work this paper cites.
A. Y. Ng, et al. , “Algorithms for inverse reinforcement learning.” in Icml , 2000, pp. 663–670
2000
Earlier work this paper cites.
P. Abbeel and A. Y. Ng, “Apprenticeship learning via inverse reinforcement learning,” in Proceedings of the twenty-first international conference on Machine learning . ACM, 2004, p. 1
2004
Earlier work this paper cites.
J. Colyar and J. Halkias, “Us highway 101 dataset,” Federal Highway Administration (FHWA), Tech. Rep. FHWA-HRT-07-030 , 2007
2007
Earlier work this paper cites.
B. D. Ziebart, et al. , “Maximum entropy inverse reinforcement learning.” in AAAI , vol. 8. Chicago, IL, USA, 2008, pp. 1433–1438
2008
Earlier work this paper cites.
B. D. Ziebart, et al. , “Navigate like a cabbie: Probabilistic reasoning from observed context-aware behavior,” in Proceedings of the 10th international conference on Ubiquitous computing . ACM, 2008, pp. 322–331
2008
Earlier work this paper cites.
D. G. R. Bradski and A. Kaehler, Learning Opencv, 1st Edition , 1st ed. O’Reilly Media, Inc., 2008
2008
Earlier work this paper cites.
B. D. Argall, et al. , “A survey of robot learning from demonstration,” Robotics and autonomous systems , vol. 57, no. 5, pp. 469–483, 2009
2009
Earlier work this paper cites.
P. Henry, et al. , “Learning to navigate through crowded environments,” in Robotics and Automation (ICRA), 2010 IEEE International Conference on . IEEE, 2010, pp. 981–986
2010
Earlier work this paper cites.
S. Ross and D. Bagnell, “Efficient reductions for imitation learning,” in Proceedings of the thirteenth international conference on artificial intelligence and statistics , 2010, pp. 661–668
2010
Earlier work this paper cites.
S. Ross, et al. , “A reduction of imitation learning and structured prediction to no-regret online learning,” in Proceedings of the fourteenth international conference on artificial intelligence and statistics , 2011, pp. 627–635
2011
Earlier work this paper cites.
D. Vasquez, et al. , “Inverse reinforcement learning algorithms and features for robot navigation in crowds: an experimental comparison,” in IEEE-RSJ Int. Conf. on Intelligent Robots and Systems , 2014, pp. 1341–1346
2014
Earlier work this paper cites.
I. Goodfellow, et al. , “Generative adversarial nets,” in Advances in neural information processing systems , 2014, pp. 2672–2680
2014
Earlier work this paper cites.
T.-Y. Lin, et al. , “Microsoft coco: Common objects in context,” in k , 2014
2014
Earlier work this paper cites.
R. Girshick, “Fast R-CNN,” in Proceedings of the International Conference on Computer Vision (ICCV) , 2015
2015
Earlier work this paper cites.
S. Ren, et al. , “Faster R-CNN: Towards real-time object detection with region proposal networks,” in Neural Information Processing Systems (NIPS) , 2015
2015
Earlier work this paper cites.
J. Schulman, et al. , “Trust region policy optimization,” in International Conference on Machine Learning , 2015, pp. 1889–1897
2015
Earlier work this paper cites.

Similar

J. Ho and S. Ermon, “Generative adversarial imitation learning,” in Advances in Neural Information Processing Systems , 2016, pp. 4565–4573
2016
Cited alongside, same era.
K. He, et al. , “Deep residual learning for image recognition,” 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 770–778, 2016
2016
Cited alongside, same era.
W. Liu, et al. , “Ssd: Single shot multibox detector,” in ECCV , 2016
2016
Cited alongside, same era.
L. Leal-Taixé, et al. , “Learning by tracking: Siamese cnn for robust target association,” 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) , pp. 418–425, 2016
2016
Cited alongside, same era.
L. Bertinetto, et al. , “Fully-convolutional siamese networks for object tracking,” in ECCV Workshops , 2016
2016
Cited alongside, same era.
A. Bewley, et al. , “Simple online and realtime tracking,” in 2016 IEEE International Conference on Image Processing (ICIP) , 2016, pp. 3464–3468
2016
Cited alongside, same era.
C. Finn, et al. , “Guided cost learning: Deep inverse optimal control via policy optimization,” in International Conference on Machine Learning , 2016, pp. 49–58
2016
Cited alongside, same era.
E. Ristani, et al. , “Performance measures and a data set for multi-target, multi-camera tracking,” in ECCV Workshops , 2016
2016
Cited alongside, same era.
K. Shiarlis, et al. , “Inverse reinforcement learning from failure,” in Proceedings of the 2016 International Conference on Autonomous Agents & Multiagent Systems , 2016, pp. 1060–1068
2016
Cited alongside, same era.
T. Zhang, et al. , “Deep imitation learning for complex manipulation tasks from virtual reality teleoperation,” arXiv preprint arXiv:1710.04615 , 2017
Original
2017
Cited alongside, same era.
K. He, et al. , “Mask R-CNN,” in Proceedings of the International Conference on Computer Vision (ICCV) , 2017
2017
Cited alongside, same era.
T. Lin, et al. , “Focal loss for dense object detection,” in IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017 , 2017, pp. 2999–3007
2017
Cited alongside, same era.

Then

N. Lee, et al. , “Desire: Distant future prediction in dynamic scenes with interacting agents,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2017, pp. 336–345
2017
Later among the works it cites.
A. Kuefler, et al. , “Imitating driver behavior with generative adversarial networks,” in Intelligent Vehicles Symposium (IV), 2017 IEEE . IEEE, 2017, pp. 204–211
2017
Later among the works it cites.
J. K. Gupta, et al. , “Cooperative multi-agent control using deep reinforcement learning,” in International Conference on Autonomous Agents and Multiagent Systems . Springer, 2017, pp. 66–83
2017
Later among the works it cites.
J. Schulman et al. , “Proximal policy optimization algorithms,” arXiv preprint arXiv:1707.06347 , 2017
Original
2017
Later among the works it cites.
Z. Wang, et al. , “Robust imitation of diverse behaviors,” in Advances in Neural Information Processing Systems , 2017, pp. 5320–5329
2017
Later among the works it cites.
T. Osa, et al. , “An algorithmic perspective on imitation learning,” Foundations and Trends® in Robotics , vol. 7, no. 1-2, pp. 1–179, 2018
2018
Closest in time.
R. Girshick, et al. , “Detectron,” https://github.com/facebookresearch/detectron , 2018
2018
Closest in time.
J. Redmon and A. Farhadi, “Yolov3: An incremental improvement,” CoRR , vol. abs/1804.02767, 2018. [Online]. Available: http://arxiv.org/abs/1804.02767
Original
2018
Closest in time.
N. Wojke and A. Bewley, “Deep cosine metric learning for person re-identification,” in 2018 IEEE Winter Conference on Applications of Computer Vision (WACV) . IEEE, 2018, pp. 748–756
2018
Closest in time.
M.-C. Chang, et al. , “Video analytics in smart transportation for the aic’18 challenge,” in CVPR Workshop (CVPRW) on the AI City Challenge , 2018
2018
Closest in time.
E. Zhan, et al. , “Generative multi-agent behavioral cloning,” arXiv preprint arXiv:1803.07612 , 2018
Original
2018
Closest in time.
X. B. Peng, et al. , “Deepmimic: Example-guided deep reinforcement learning of physics-based character skills,” arXiv preprint arXiv:1804.02717 , 2018
Original
2018
Closest in time.
R. P. Bhattacharyya, et al. , “Multi-agent imitation learning for driving simulation,” arXiv preprint arXiv:1803.01044 , 2018
Original
2018
Closest in time.
Y. Aytar, et al. , “Playing hard exploration games by watching youtube,” arXiv preprint arXiv:1805.11592 , 2018
Original
2018
Closest in time.
X. B. Peng, et al. , “Sfv: Reinforcement learning of physical skills from videos,” arXiv preprint arXiv:1810.03599 , 2018
Original
2018
Closest in time.
J. Song, et al. , “Multi-agent generative adversarial imitation learning,” arXiv , 2018
2018
Closest in time.
OpenAI, “Openai five,” 2018. [Online]. Available: https://blog.openai.com/openai-five/
2018
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…