2017

ConvNets and ImageNet Beyond Accuracy: Understanding Mistakes and Uncovering Biases

Stock, Pierre, Cisse, Moustapha

Understand

ConvNets and Imagenet have driven the recent success of deep learning for image classification.

However, the marked slowdown in performance improvement combined with the lack of robustness of neural networks to adversarial examples and their tendency to exhibit undesirable biases question the reliability of these methods.
This work investigates these questions from the perspective of the end-user by using human subject studies and explanations.
The contribution of this study is threefold.

Built on

Human problem solving: The state of the theory in 1970 (1972)
Simon, H.A., Newell, A.: · 1972
Earlier work this paper cites.
Case-based reasoning; foundational issues, methodological variations, and system approaches
Aamodt, A., Plaza, E.: · 1994
Earlier work this paper cites.
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., Haffner, P.: · 1998
Earlier work this paper cites.
Case-based reasoning in the health sciences: What’s next?
Bichindaritz, I., Marling, C.: · 2006
Earlier work this paper cites.
Bayesian data analysis using r
Gelman, A.: · 2006
Earlier work this paper cites.
A kernel method for the two-sample-problem
Gretton, A., Borgwardt, K.M., Rasch, M., Schölkopf, B., Smola, A.J.: · 2006
Earlier work this paper cites.
ImageNet: A Large-Scale Hierarchical Image Database
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: · 2009
Earlier work this paper cites.
Improving the fisher kernel for large-scale image classification
Perronnin, F., Sánchez, J., Mensink, T.: · 2010
Earlier work this paper cites.
Imagenet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., Hinton, G.E.: · 2012
Earlier work this paper cites.
Intriguing properties of neural networks
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I.J., Fergus, R.: · 2013
Earlier work this paper cites.
Adam: A method for stochastic optimization
Kingma, D.P., Ba, J.: · 2014
Earlier work this paper cites.
Explaining and harnessing adversarial examples
Goodfellow, I.J., Shlens, J., Szegedy, C.: · 2014
Earlier work this paper cites.
What i learned from competing against a convnet on imagenet, http://karpathy.github.io/2014/ 09/02/what-i-learned-from-competing-against-a-convnet-on-imagenet/
Karpathy, A.: · 2014
Earlier work this paper cites.
The bayesian case model: A generative approach for case-based reasoning and prototype classification
Kim, B., Rudin, C., Shah, J.A.: · 2014
Earlier work this paper cites.
Streaming submodular maximization: Massive data summarization on the fly
Badanidiyuru, A., Mirzasoleiman, B., Karbasi, A., Krause, A.: · 2014
Earlier work this paper cites.

Similar

Corpus-based estimates of word association predict biases in judgment of word co-occurrence likelihood
Paperno, D., Marelli, M., Tentori, K., Baroni, M.: · 2014
Cited alongside, same era.
Fast r-cnn
Girshick, R.: · 2015
Cited alongside, same era.
Explaining and harnessing adversarial examples
Goodfellow, I., Shlens, J., Szegedy, C.: · 2015
Cited alongside, same era.
Exploring the space of adversarial images
Tabacof, P., Valle, E.: · 2015
Cited alongside, same era.
Analysis of classifiers’ robustness to adversarial perturbations
Fawzi, A., Fawzi, O., Frossard, P.: · 2015
Cited alongside, same era.
Understanding adversarial training: Increasing local stability of neural nets through robust optimization
Shaham, U., Yamada, Y., Negahban, S.: · 2015
Cited alongside, same era.
Deepfool: a simple and accurate method to fool deep neural networks
Moosavi-Dezfooli, S., Fawzi, A., Frossard, P.: · 2015
Cited alongside, same era.
Deep residual learning for image recognition
He, K., Zhang, X., Ren, S., Sun, J.: · 2016
Cited alongside, same era.
Densely connected convolutional networks
Huang, G., Liu, Z., Weinberger, K.Q.: · 2016
Cited alongside, same era.
Deepercut: A deeper, stronger, and faster multi-person pose estimation model
Original
Insafutdinov, E., Pishchulin, L., Andres, B., Andriluka, M., Schiele, B.: · 2016
Cited alongside, same era.
Human pose estimation via convolutional part heatmap regression
Bulat, A., Tzimiropoulos, G.: · 2016
Cited alongside, same era.
Man is to computer programmer as woman is to homemaker? debiasing word embeddings
Bolukbasi, T., Chang, K., Zou, J.Y., Saligrama, V., Kalai, A.: · 2016
Cited alongside, same era.

Then

Adversarial examples in the physical world
Kurakin, A., Goodfellow, I.J., Bengio, S.: · 2016
Later among the works it cites.
The mythos of model interpretability
Lipton, Z.C.: · 2016
Later among the works it cites.
”why should i trust you?”: Explaining the predictions of any classifier
Ribeiro, M.T., Singh, S., Guestrin, C.: · 2016
Later among the works it cites.
Mask R-CNN
Original
He, K., Gkioxari, G., Dollár, P., Girshick, R.B.: · 2017
Closest in time.
Convolutional sequence to sequence learning
Gehring, J., Auli, M., Grangier, D., Yarats, D., Dauphin, Y.N.: · 2017
Closest in time.
Residual convolutional CTC networks for automatic speech recognition
Wang, Y., Deng, X., Pu, S., Huang, Z.: · 2017
Closest in time.
mixup: Beyond empirical risk minimization
Zhang, H., Cissé, M., Dauphin, Y.N., Lopez-Paz, D.: · 2017
Closest in time.
Houdini: Fooling deep structured visual and speech recognition models with adversarial examples
Cissé, M., Adi, Y., Neverova, N., Keshet, J.: · 2017
Closest in time.
Cognitive psychology for deep neural networks: A shape bias case study
Ritter, S., Barrett, D.G.T., Santoro, A., Botvinick, M.M.: · 2017
Closest in time.
Parseval networks: Improving robustness to adversarial examples
Cisse, M., Bojanowski, P., Grave, E., Dauphin, Y., Usunier, N.: · 2017
Closest in time.
Countering adversarial images using input transformations
Guo, C., Rana, M., Cissé, M., van der Maaten, L.: · 2017
Closest in time.
Explainable artificial intelligence: Understanding, visualizing and interpreting deep learning models
Samek, W., Wiegand, T., Müller, K.: · 2017
Closest in time.
Methods for interpreting and understanding deep neural networks
Montavon, G., Samek, W., Müller, K.: · 2017
Closest in time.
Towards interpretable deep neural networks by leveraging adversarial examples
Dong, Y., Su, H., Zhu, J., Bao, F.: · 2017
Closest in time.
Ensemble adversarial training: Attacks and defenses
Kurakin, A., Boneh, D., Tramèr, F., Goodfellow, I., Papernot, N., McDaniel, P.: · 2018
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…