2020

Selectivity considered harmful: evaluating the causal impact of class selectivity in DNNs

Leavitt, Matthew L., Morcos, Ari

Understand

The properties of individual neurons are often analyzed in order to understand the biological and artificial neural networks in which they're embedded.

Class selectivity-typically defined as how different a neuron's responses are across different classes of stimuli or data samples-is commonly used for this purpose.
However, it remains an open question whether it is necessary and/or sufficient for deep neural networks (DNNs) to learn class selectivity in individual units.
We investigated the causal impact of class selectivity on network function by directly regularizing for or against class selectivity.

Built on

Ablation Studies in Artificial Neural Networks
Original
Richard Meyes, Melanie Lu, Constantin Waubert de Puiseau, and Tobias Meisen · 1901
Earlier work this paper cites.
Dying ReLU and Initialization: Theory and Numerical Examples
Original
Lu Lu, Yeonjong Shin, Yanhui Su, and George Em Karniadakis · 1903
Earlier work this paper cites.
The integrative action of the nervous system
Charles S. Sherrington · 1906
Earlier work this paper cites.
SciPy 1.0–Fundamental Algorithms for Scientific Computing in Python
Original
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C. J. Carey, Ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1 0 Contributors · 1907
Earlier work this paper cites.
The impulses produced by sensory nerve endings
E. D. Adrian · 1926
Earlier work this paper cites.
Relations Between Two Sets of Variates
Harold Hotelling · 1936
Earlier work this paper cites.
Receptors and sensory perception
Ragnar Granit · 1955
Earlier work this paper cites.
Receptive fields of single neurones in the cat’s striate cortex
D. H. Hubel and T. N. Wiesel · 1959
Earlier work this paper cites.
Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex
D. H. Hubel and T. N. Wiesel · 1962
Earlier work this paper cites.
The hippocampus as a spatial map. Preliminary evidence from unit activity in the freely-moving rat
J. O’Keefe and J. Dostrovsky · 1971
Earlier work this paper cites.
Single Units and Sensation: A Neuron Doctrine for Perceptual Psychology?
H B Barlow · 1972
Earlier work this paper cites.
Exploration of the primary visual cortex, 1955–78
David H. Hubel · 1982
Earlier work this paper cites.
Postnatal development of the visual cortex and the influence of environment
T. N. Wiesel · 1982
Earlier work this paper cites.
A Model That Accounts for Activity in Primate Frontal Cortex during a Delayed Matching-to-Sample Task
Sohie Lee Moody, Steven P. Wise, Giuseppe di Pellegrino, and David Zipser · 1998
Earlier work this paper cites.
Principles of neural science
E R Kandel, J H Schwartz, and Jessica Chao · 2000
Earlier work this paper cites.
Deleting object selective units in a fully-connected layer of deep convolutional networks improves classification performance
Original
Yuta Kanda, Kota S. Sasaki, Izumi Ohzawa, and Hiroshi Tamura · 2001
Earlier work this paper cites.
Spatial Representation in the Entorhinal Cortex
Marianne Fyhn, Sturla Molden, Menno P. Witter, Edvard I. Moser, and May-Britt Moser · 2004
Earlier work this paper cites.
Visualizing Higher-Layer Features of a Deep Network
Dumitru Erhan, Yoshua Bengio, Aaron C. Courville, and Pascal Vincent · 2009
Earlier work this paper cites.
Learning Multiple Layers of Features from Tiny Images
Alex Krizhevsky · 2009
Earlier work this paper cites.
Rectifier nonlinearities improve neural network acoustic models
Andrew L. Maas, Awni Y. Hannun, and Andrew Y. Ng · 2013
Earlier work this paper cites.
Cortical control of arm movements: a dynamical systems perspective
Krishna V Shenoy, Maneesh Sahani, and Mark M Churchland · 2013
Earlier work this paper cites.
A category-free neural population supports evolving demands during decision-making
David Raposo, Matthew T Kaufman, and Anne K Churchland · 2014
Earlier work this paper cites.
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Original
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman · 2014
Earlier work this paper cites.

Similar

Visualizing and Understanding Convolutional Networks
Matthew D. Zeiler and Rob Fergus · 2014
Cited alongside, same era.
Tiny imagenet visual recognition challenge, 2015
Li Fei-Fei, Andrej Karpathy, and Justin Johnson · 2015
Cited alongside, same era.
Adam: A method for stochastic optimization
Original
Diederik P. Kingma and Jimmy Ba · 2015
Cited alongside, same era.
Convergent learning: Do different neural networks learn the same representations?
Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John Hopcroft · 2015
Cited alongside, same era.
Very deep convolutional networks for large-scale image recognition
Original
Karen Simonyan and Andrew Zisserman · 2015
Cited alongside, same era.
A positive-negative mode of population covariation links brain connectivity, demographics and behavior
Stephen M. Smith, Thomas E. Nichols, Diego Vidaurre, Anderson M. Winkler, Timothy E. J. Behrens, Matthew F. Glasser, Kamil Ugurbil, Deanna M. Barch, David C. Van Essen, and Karla L. Miller · 2015
Cited alongside, same era.
A neural network that finds a naturalistic solution for the production of muscle activity
David Sussillo, Mark M Churchland, Matthew T Kaufman, and Krishna V Shenoy · 2015
Cited alongside, same era.
Understanding Neural Networks Through Deep Visualization
Original
Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson · 2015
Cited alongside, same era.
Object Detectors Emerge in Deep Scene CNNs
Original
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba · 2015
Cited alongside, same era.
Why neurons mix: high dimensionality for higher cognition
Stefano Fusi, Earl K. Miller, and Mattia Rigotti · 2016
Cited alongside, same era.
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2016
Cited alongside, same era.
Visualizing and Understanding Recurrent Networks
Andrej Karpathy, Justin Johnson, and Li Fei-Fei · 2016
Cited alongside, same era.

Then

Ablation of a Robot’s Brain: Neural Networks Under a Knife
Original
Peter E. Lillian, Richard Meyes, and Tobias Meisen · 2018
Later among the works it cites.
The Building Blocks of Interpretability
Chris Olah, Arvind Satyanarayan, Ian Johnson, Shan Carter, Ludwig Schubert, Katherine Ye, and Alexander Mordvintsev · 2018
Later among the works it cites.
Causal importance of orientation selectivity for generalization in image recognition
Jumpei Ukita · 2018
Later among the works it cites.
Interpreting Deep Visual Representations via Network Dissection
B. Zhou, D. Bau, A. Oliva, and A. Torralba · 2018
Later among the works it cites.
Revisiting the Importance of Individual Units in CNNs via Ablation
Original
Bolei Zhou, Yiyou Sun, David Bau, and Antonio Torralba · 2018
Later among the works it cites.
The role of untuned neurons in sensory information coding
Joel Zylberberg · 2018
Later among the works it cites.
What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Models
Fahim Dalvi, Nadir Durrani, Hassan Sajjad, Yonatan Belinkov, Anthony Bau, and James Glass · 2019
Later among the works it cites.
How Important is a Neuron
Kedar Dhamdhere, Mukund Sundararajan, and Qiqi Yan · 2019
Later among the works it cites.
On Interpretability and Feature Representations: An Analysis of the Sentiment Neuron
Jonathan Donnelly and Adam Roegiest · 2019
Later among the works it cites.
Selectivity metrics provide misleading estimates of the selectivity of single units in neural networks
Ella Gale, Ryan Blything, Nicholas Martin, Jeffrey S. Bowers, and Anh Nguyen · 2019
Later among the works it cites.
Oscillatory recurrent gated neural integrator circuits (organics), a unifying theoretical framework for neural dynamics
David J. Heeger and Wayne E. Mackey · 2019
Later among the works it cites.
A Benchmark for Interpretability Methods in Deep Neural Networks
Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim · 2019
Later among the works it cites.
Trivializations for gradient-based optimization on manifolds
Mario Lezcano-Casado · 2019
Later among the works it cites.
Discovery of Natural Language Concepts in Individual Units of CNNs
Seil Na, Yo Joong Choe, Dong-Hyun Lee, and Gunhee Kim · 2019
Later among the works it cites.
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala · 2019
Later among the works it cites.
The language of the brain: real-world neural population codes
J Andrew Pruszynski and Joel Zylberberg · 2019
Later among the works it cites.
Understanding trained CNNs by indexing neuron selectivity
Original
Ivet Rafegas, Maria Vanrell, Luis A. Alexandre, and Guillem Arias · 2019
Later among the works it cites.
Towards the neural population doctrine
Shreya Saxena and John P Cunningham · 2019
Later among the works it cites.
Understanding the role of individual units in a deep neural network
David Bau, Jun-Yan Zhu, Hendrik Strobelt, Agata Lapedriza, Bolei Zhou, and Antonio Torralba · 2020
Closest in time.
Zoom In: An Introduction to Circuits
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter · 2020
Closest in time.
Cortical population activity within a preserved neural manifold underlies multiple motor behaviors
Juan A. Gallego, Matthew G. Perich, Stephanie N. Naufel, Christian Ethier, Sara A. Solla, and Lee E. Miller · 2041
Closest in time.
Spike-timing-dependent ensemble encoding by non-classically responsive cortical neurons
Michele N Insanally, Ioana Carcea, Rachel E Field, Chris C Rodgers, Brian DePasquale, Kanaka Rajan, Michael R DeWeese, Badr F Albanna, and Robert C Froemke · 2050
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…