2020

Better Aggregation in Test-Time Augmentation

Shanmugam, Divya, Blalock, Davis, Balakrishnan, Guha et al.

Understand

Test-time augmentation -- the aggregation of predictions across transformed versions of a test input -- is a common practice in image classification.

Traditionally, predictions are combined using a simple average.
In this paper, we present 1) experimental analyses that shed light on cases in which the simple average is suboptimal and 2) a method to address these shortcomings.
A key finding is that even when test-time augmentation produces a net improvement in accuracy, it can change many correct predictions into incorrect predictions.

Built on

Imagenet: A large-scale hierarchical image database
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei · 2009
Earlier work this paper cites.
Learning multiple layers of features from tiny images
Alex Krizhevsky, Geoffrey Hinton, et al · 2009
Earlier work this paper cites.
An analysis of single-layer networks in unsupervised feature learning
Adam Coates, Andrew Ng, and Honglak Lee · 2011
Earlier work this paper cites.
Imagenet classification with deep convolutional neural networks
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton · 2012
Earlier work this paper cites.
Some improvements on deep convolutional neural network based image classification
Original
Andrew G Howard · 2013
Earlier work this paper cites.
Explaining and harnessing adversarial examples
Original
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy · 2014
Earlier work this paper cites.
Very deep convolutional networks for large-scale image recognition
Original
Karen Simonyan and Andrew Zisserman · 2014
Earlier work this paper cites.
Apac: Augmented pattern classification with neural networks
Original
Ikuro Sato, Hiroki Nishimura, and Kensuke Yokoi · 2015
Earlier work this paper cites.
Going deeper with convolutions
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich · 2015
Earlier work this paper cites.
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2016
Earlier work this paper cites.
A baseline for detecting misclassified and out-of-distribution examples in neural networks
Original
Dan Hendrycks and Kevin Gimpel · 2016
Earlier work this paper cites.
Deepfool: a simple and accurate method to fool deep neural networks
Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard · 2016
Earlier work this paper cites.

Similar

Rethinking the inception architecture for computer vision
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna · 2016
Cited alongside, same era.
On calibration of modern neural networks
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger · 2017
Cited alongside, same era.
Image classification of melanoma, nevus and seborrheic keratosis by deep neural network ensemble
Original
Kazuhisa Matsunaga, Akira Hamada, Akane Minagawa, and Hiroshi Koga · 2017
Cited alongside, same era.
Automatic differentiation in pytorch
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer · 2017
Cited alongside, same era.
Pixeldefend: Leveraging generative models to understand and defend against adversarial examples
Original
Yang Song, Taesup Kim, Sebastian Nowozin, Stefano Ermon, and Nate Kushman · 2017
Cited alongside, same era.
Test-time data augmentation for estimation of heteroscedastic aleatoric uncertainty in deep neural networks
Murat Seckin Ayhan and Philipp Berens · 2018
Cited alongside, same era.
A deep 3d residual cnn for false-positive reduction in pulmonary nodule detection
Hongsheng Jin, Zongyao Li, Ruofeng Tong, and Lanfen Lin · 2018
Cited alongside, same era.
Deflecting adversarial attacks with pixel deflection
Aaditya Prakash, Nick Moran, Solomon Garber, Antonella DiLillo, and James Storer · 2018
Cited alongside, same era.
Mobilenetv2: Inverted residuals and linear bottlenecks
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen · 2018
Cited alongside, same era.
Understanding measures of uncertainty for adversarial example detection
Original
Lewis Smith and Yarin Gal · 2018
Cited alongside, same era.
Pretrained models for pytorch
Remi Cadene · 2019
Cited alongside, same era.

Then

Certified adversarial robustness via randomized smoothing
Original
Jeremy M Cohen, Elan Rosenfeld, and J Zico Kolter · 2019
Later among the works it cites.
Autoaugment: Learning augmentation strategies from data
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le · 2019
Later among the works it cites.
cnn-finetune
Alex Parinov · 2019
Later among the works it cites.
One pixel attack for fooling deep neural networks
Jiawei Su, Danilo Vasconcellos Vargas, and Kouichi Sakurai · 2019
Later among the works it cites.
Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks
Guotai Wang, Wenqi Li, Michael Aertsen, Jan Deprest, Sébastien Ourselin, and Tom Vercauteren · 2019
Later among the works it cites.
Pretrained models for pytorch
Aaron Xichen · 2019
Later among the works it cites.
Classification confidence estimation with test-time data-augmentation
Original
Yuval Bahat and Gregory Shakhnarovich · 2020
Closest in time.
Learning invariances in neural networks
Original
Gregory Benton, Marc Finzi, Pavel Izmailov, and Andrew Gordon Wilson · 2020
Closest in time.
Are we done with imagenet?
Original
Lucas Beyer, Olivier J Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord · 2020
Closest in time.
Learning loss for test-time augmentation
Ildoo Kim, Younghoon Kim, and Sungwoong Kim · 2020
Closest in time.
Greedy policy search: A simple baseline for learnable test-time augmentation
Original
Dmitry Molchanov, Alexander Lyzhov, Yuliya Molchanova, Arsenii Ashukha, and Dmitry Vetrov · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…