2018

Structured Domain Randomization: Bridging the Reality Gap by Context-Aware Synthetic Data

Prakash, Aayush, Boochoon, Shaad, Brophy, Mark et al.

Understand

We present structured domain randomization (SDR), a variant of domain randomization (DR) that takes into account the structure and context of the scene.

In contrast to DR, which places objects and distractors randomly according to a uniform probability distribution, SDR places objects and distractors randomly according to probability distributions that arise from the specific problem at hand.
In this manner, SDR-generated imagery enables the neural network to take the context around an object into consideration during detection.
We demonstrate the power of SDR for the problem of 2D bounding box car detection, achieving competitive results on real data after training only on synthetic data.

Built on

N. Qian, “On the momentum term in gradient descent learning algorithms,”
1999
Earlier work this paper cites.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “ImageNet: A large-scale hierarchical image database,” in
2009
Earlier work this paper cites.
D. J. Butler, J. Wulff, G. B. Stanley, and M. J. Black, “A naturalistic open source movie for optical flow evaluation,” in
2012
Earlier work this paper cites.
A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for autonomous driving? The KITTI vision benchmark suite,” in
2012
Earlier work this paper cites.
M. Hodosh, P. Young, and J. Hockenmaier, “Framing image description as a ranking task: Data, models and evaluation metrics,”
2013
Earlier work this paper cites.
T. Lin, M. Maire, S. J. Belongie, L. D. Bourdev, R. B. Girshick, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft COCO: Common objects in context,” in
2014
Earlier work this paper cites.
A. Dosovitskiy, P. Fischer, E. Ilg, P. Häusser, C. Hazırbaş, V. Golkov, P. Smagt, D. Cremers, and T. Brox, “FlowNet: Learning optical flow with convolutional networks,” in
2015
Earlier work this paper cites.
Y. Zhu, R. Urtasun, R. Salakhutdinov, and S. Fidler, “segDeepM: Exploiting segmentation and context in deep neural networks for object detection,” in
2015
Earlier work this paper cites.
S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: Towards real-time object detection with region proposal networks,” in
2015
Earlier work this paper cites.
A. Handa, V. Pătrăucean, V. Badrinarayanan, S. Stent, and R. Cipolla, “SceneNet: Understanding real world indoor scenes with synthetic data,” in
2016
Earlier work this paper cites.
N. Mayer, E. Ilg, P. Hausser, P. Fischer, D. Cremers, A. Dosovitskiy, and T. Brox, “A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation,” in
2016
Earlier work this paper cites.

Similar

W. Qiu and A. Yuille, “UnrealCV: Connecting computer vision to Unreal Engine,” in
Original
2016
Cited alongside, same era.
Y. Zhang, W. Qiu, Q. Chen, X. Hu, and A. Yuille, “UnrealStereo: A synthetic dataset for analyzing stereo vision,” in
Original
2016
Cited alongside, same era.
G. Ros, L. Sellart, J. Materzynska, D. Vazquez, and A. Lopez, “The SYNTHIA dataset: A large collection of synthetic images for semantic segmentation of urban scenes,” in
2016
Cited alongside, same era.
S. R. Richter, V. Vineet, S. Roth, and V. Koltun, “Playing for data: Ground truth from computer games,” in
2016
Cited alongside, same era.
A. Gaidon, Q. Wang, Y. Cabon, and E. Vig, “Virtual worlds as proxy for multi-object tracking analysis,” in
2016
Cited alongside, same era.
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in
2016
Cited alongside, same era.
J. McCormac, A. Handa, and S. Leutenegger, “SceneNet RGB-D: 5M photorealistic images of synthetic indoor trajectories with ground truth,” in
2017
Cited alongside, same era.
M. Mueller, V. Casser, J. Lahoud, N. Smith, and B. Ghanem, “Sim4CV: A photo-realistic simulator for computer vision applications,” in
Original
2017
Cited alongside, same era.
A. Tsirikoglou, J. Kronander, M. Wrenninge, and J. Unger, “Procedural modeling and physically based rendering for synthetic data generation in automotive applications,” in
Original
2017
Cited alongside, same era.
J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel, “Domain randomization for transferring deep neural networks from simulation to the real world,” in
2017
Cited alongside, same era.
M. Johnson-Roberson, C. Barto, R. Mehta, S. N. Sridhar, K. Rosaen, and R. Vasudevan, “Driving in the matrix: Can virtual worlds replace human-generated annotations for real world tasks?” in
2017
Cited alongside, same era.

Then

S. Hinterstoisser, V. Lepetit, P. Wohlhart, and K. Konolige, “On pre-trained image features and synthetic images for deep learning,” in
Original
2017
Later among the works it cites.
D. Dwibedi, I. Misra, and M. Hebert, “Cut, paste and learn: Surprisingly easy synthesis for instance detection,” in
2017
Later among the works it cites.
G. Georgakis, A. Mousavian, A. C. Berg, and J. Kosecka, “Synthesizing training data for object detection in indoor scenes,” in
2017
Later among the works it cites.
S. R. Richter, Z. Hayder, and V. Koltun, “Playing for benchmarks,” in
2017
Later among the works it cites.
J. Tremblay, A. Prakash, D. Acuna, M. Brophy, V. Jampani, C. Anil, T. To, E. Cameracci, S. Boochoon, and S. Birchfield, “Training deep networks with synthetic data: Bridging the reality gap by domain randomization,” in
2018
Closest in time.
F. Yu, W. Xian, Y. Chen, F. Liu, M. Liao, V. Madhavan, and T. Darrell, “Bdd100k: A diverse driving video database with scalable annotation tooling,”
Original
2018
Closest in time.
J. Borrego, A. Dehban, R. Figueiredo, P. Moreno, A. Bernardino, and J. Santos-Victor, “Applying domain randomization to synthetic data for object category detection,”
Original
2018
Closest in time.
S. Suwajanakorn, N. Snavely, J. Tompson, and M. Norouzi, “Discovery of latent 3D keypoints via end-to-end geometric reasoning,”
2018
Closest in time.
N. Mayer, E. Ilg, P. Fischer, C. Hazirbas, D. Cremers, A. Dosovitskiy, and T. Brox, “What makes good synthetic training data for learning disparity and optical flow estimation?”
2018
Closest in time.
M. Sundermeyer, Z.-C. Marton, M. Durner, M. Brucker, and R. Triebel, “Implicit 3D orientation learning for 6D object detection from RGB images,” in
2018
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…