2020

When and how CNNs generalize to out-of-distribution category-viewpoint combinations

Madan, Spandan, Henry, Timothy, Dozier, Jamell et al.

Understand

Object recognition and viewpoint estimation lie at the heart of visual understanding.

Recent works suggest that convolutional neural networks (CNNs) fail to generalize to out-of-distribution (OOD) category-viewpoint combinations, ie.
combinations not seen during training.
In this paper, we investigate when and how such OOD generalization may be possible by evaluating CNNs trained to classify both object category and 3D viewpoint on OOD combinations, and identifying the neural mechanisms that facilitate such OOD generalization.

Built on

Learning, invariance, and generalization in high-order neural networks
C Lee Giles and Tom Maxwell · 1987
Earlier work this paper cites.
A neurobiological model of visual attention and invariant pattern recognition based on dynamic routing of information
Bruno A Olshausen, Charles H Anderson, and David C Van Essen · 1993
Earlier work this paper cites.
3D object recognition: A model of view-tuned neurons
Emanuela Bricolo, Tomaso Poggio, and Nikos K Logothetis · 1997
Earlier work this paper cites.
Multitask learning
Rich Caruana · 1997
Earlier work this paper cites.
Gradient-based learning applied to document recognition
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner · 1998
Earlier work this paper cites.
Just one view: Invariances in inferotemporal cell tuning
Maximilian Riesenhuber and Tomaso Poggio · 1998
Earlier work this paper cites.
Invariant visual representation by single neurons in the human brain
R Quian Quiroga, Leila Reddy, Gabriel Kreiman, Christof Koch, and Itzhak Fried · 2005
Earlier work this paper cites.
An empirical evaluation of deep architectures on problems with many factors of variation
Hugo Larochelle, Dumitru Erhan, Aaron Courville, James Bergstra, and Yoshua Bengio · 2007
Earlier work this paper cites.
3d generic object categorization, localization and pose estimation
Silvio Savarese and Li Fei-Fei · 2007
Earlier work this paper cites.
Measuring invariances in deep networks
Ian Goodfellow, Honglak Lee, Quoc V Le, Andrew Saxe, and Andrew Y Ng · 2009
Earlier work this paper cites.
Pose estimation for category specific multiview object localization
Mustafa Ozuysal, Vincent Lepetit, and Pascal Fua · 2009
Earlier work this paper cites.
Selectivity and tolerance (“invariance”) both increase as visual information propagates from cortical area V4 to IT
Nicole C Rust and James J DiCarlo · 2010
Earlier work this paper cites.
Unbiased look at dataset bias
Antonio Torralba and Alexei A Efros · 2011
Earlier work this paper cites.
Improving object classification using pose information
Hugo Penedones, Ronan Collobert, Francois Fleuret, and David Grangier · 2012
Earlier work this paper cites.
3D object representations for fine-grained categorization
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei · 2013
Earlier work this paper cites.
Is 2D information enough for viewpoint estimation?
Amir Ghodrati, Marco Pedersoli, and Tinne Tuytelaars · 2014
Earlier work this paper cites.
Adam: A method for stochastic optimization
Original
Diederik P Kingma and Jimmy Ba · 2014
Earlier work this paper cites.
Deep inside convolutional networks: Visualising image classification models and saliency maps
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman · 2014
Earlier work this paper cites.
Beyond pascal: A benchmark for 3D object detection in the wild
Yu Xiang, Roozbeh Mottaghi, and Silvio Savarese · 2014
Earlier work this paper cites.
Visualizing and understanding convolutional networks
Matthew D Zeiler and Rob Fergus · 2014
Earlier work this paper cites.
Is object localization for free?-weakly-supervised learning with convolutional neural networks
Maxime Oquab, Léon Bottou, Ivan Laptev, and Josef Sivic · 2015
Earlier work this paper cites.
Render for CNN: Viewpoint estimation in images using CNNs trained with rendered 3D model views
Hao Su, Charles R Qi, Yangyan Li, and Leonidas J Guibas · 2015
Earlier work this paper cites.
Viewpoints and keypoints
Shubham Tulsiani and Jitendra Malik · 2015
Earlier work this paper cites.

Similar

Pose induction for novel object categories
Shubham Tulsiani, Joao Carreira, and Jitendra Malik · 2015
Cited alongside, same era.
Object detectors emerge in deep scene CNNs
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba · 2015
Cited alongside, same era.
iLab-20M: A large-scale controlled object dataset to investigate deep learning
Ali Borji, Saeed Izadi, and Laurent Itti · 2016
Cited alongside, same era.
A comparative analysis and study of multiview CNN models for joint object categorization and pose estimation
Mohamed Elhoseiny, Tarek El-Gaaly, Amr Bakry, and Ahmed Elgammal · 2016
Cited alongside, same era.
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2016
Cited alongside, same era.
Crafting a multi-task CNN for viewpoint estimation
Francisco Massa, Renaud Marlet, and Mathieu Aubry · 2016
Cited alongside, same era.
Visual cortex and deep networks: learning invariant representations
Tomaso Poggio and Fabio Anselmi · 2016
Cited alongside, same era.
UnrealCV: Connecting computer vision to Unreal Engine
Weichao Qiu and Alan Yuille · 2016
Cited alongside, same era.
Rethinking the inception architecture for computer vision
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna · 2016
Cited alongside, same era.
Wide residual networks
Sergey Zagoruyko and Nikos Komodakis · 2016
Cited alongside, same era.
Network dissection: Quantifying interpretability of deep visual representations
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba · 2017
Cited alongside, same era.
CARLA: An open urban driving simulator
Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun · 2017
Cited alongside, same era.

Then

Convolutional networks for object category and 3D pose estimation from 2D images
Siddharth Mahendran, Haider Ali, and René Vidal · 2018
Later among the works it cites.
On the importance of single directions for generalization
Ari S Morcos, David GT Barrett, Neil C Rabinowitz, and Matthew Botvinick · 2018
Later among the works it cites.
Pixels, voxels, and views: A study of shape representations for single view 3D object shape prediction
Daeyun Shin, Charless C Fowlkes, and Derek Hoiem · 2018
Later among the works it cites.
PoseCNN: A convolutional neural network for 6d object pose estimation in cluttered scenes
Yu Xiang, Tanner Schmidt, Venkatraman Narayanan, and Dieter Fox · 2018
Later among the works it cites.
An overview of multi-task learning
Yu Zhang and Qiang Yang · 2018
Later among the works it cites.
Revisiting the importance of individual units in cnns via ablation
Original
Bolei Zhou, Yiyou Sun, David Bau, and Antonio Torralba · 2018
Later among the works it cites.
Strike (with) a pose: Neural networks are easily fooled by strange poses of familiar objects
Michael A Alcorn, Qi Li, Zhitao Gong, Chengfei Wang, Long Mai, Wei-Shinn Ku, and Anh Nguyen · 2019
Later among the works it cites.
Why do deep convolutional networks generalize so poorly to small image transformations?
Aharon Azulay and Yair Weiss · 2019
Later among the works it cites.
ObjectNet: A large-scale bias-controlled dataset for pushing the limits of object recognition models
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz · 2019
Later among the works it cites.
Gauge equivariant convolutional networks and the Icosahedral CNN
Taco S Cohen, Maurice Weiler, Berkay Kicanaoglu, and Max Welling · 2019
Later among the works it cites.
Exploring the landscape of spatial robustness
Logan Engstrom, Brandon Tran, Dimitris Tsipras, Ludwig Schmidt, and Aleksander Madry · 2019
Later among the works it cites.
Physics-based rendering for improving robustness to rain
Shirsendu Sukanta Halder, Jean-François Lalonde, and Raoul de Charette · 2019
Later among the works it cites.
Spair-71k: A large-scale benchmark for semantic correspondence
Original
Juhong Min, Jongmin Lee, Jean Ponce, and Minsu Cho · 2019
Later among the works it cites.
Minimal images in deep neural networks: Fragile object recognition in natural images
Sanjana Srivastava, Guy Ben-Yosef, and Xavier Boix · 2019
Later among the works it cites.
Task representations in neural networks trained to perform many cognitive tasks
Guangyu Robert Yang, Madhura R Joglekar, H Francis Song, William T Newsome, and Xiao-Jing Wang · 2019
Later among the works it cites.
nuScenes: A multimodal dataset for autonomous driving
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom · 2020
Closest in time.
Blender - a 3D modelling and rendering package
Blender Online Community · 2020
Closest in time.
CPS++: Improving class-level 6D pose and shape estimation from monocular images with self-supervised learning
Original
Fabian Manhardt, Gu Wang, Benjamin Busam, Manuel Nickel, Sven Meier, Luca Minciullo, Xiangyang Ji, and Nassir Navab · 2020
Closest in time.
Esri CityEngine - a 3D city modeling software for urban design, visual effects, and VR/AR
Pascal Mueller, Simon Haegler, Andreas Ulmer, Matthias Schubiger, Stefan Müller Arisona, and Basil Weber · 2020
Closest in time.
Deep double descent: Where bigger models and more data hurt
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever · 2020
Closest in time.
Which tasks should be learned together in multi-task learning?
Trevor Standley, Amir R Zamir, Dawn Chen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese · 2020
Closest in time.
Frivolous units: Wider networks are not really that wide
Stephen Casper, Xavier Boix, Vanessa D’Amario, Ling Guo, Martin Schrimpf, Kasper Vinken, and Gabriel Kreiman · 2021
Closest in time.
A survey on multi-task learning
Yu Zhang and Qiang Yang · 2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…