2023

Towards Stable Test-Time Adaptation in Dynamic Wild World

Niu, Shuaicheng, Wu, Jiaxiang, Zhang, Yifan et al.

Understand

Test-time adaptation (TTA) has shown to be effective at tackling distribution shifts between training and testing data by adapting a given model on test samples.

However, the online model updating of TTA may be unstable and this is often a key obstacle preventing existing TTA methods from being deployed in the real world.
Specifically, TTA may fail to improve or even harm the model performance when test data have: 1) mixed distribution shifts, 2) small batch sizes, and 3) online imbalanced label distribution shifts, which are quite common in practice.
In this paper, we investigate the unstable reasons and find that the batch norm layer is a crucial factor hindering TTA stability.

Built on

Flat minima
Sepp Hochreiter and Jürgen Schmidhuber · 1997
Earlier work this paper cites.
COVID-DA: Deep domain adaptation from typical pneumonia to covid-19
Original
Yifan Zhang, Shuaicheng Niu, Zhen Qiu, Ying Wei, Peilin Zhao, Jianhua Yao, Junzhou Huang, Qingyao Wu, and Mingkui Tan · 2005
Earlier work this paper cites.
Imagenet: A large-scale hierarchical image database
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei · 2009
Earlier work this paper cites.
Online learning: Random averages, combinatorial parameters, and learnability
Alexander Rakhlin, Karthik Sridharan, and Ambuj Tewari · 2010
Earlier work this paper cites.
Double updating online learning
Peilin Zhao, Steven CH Hoi, and Rong Jin · 2011
Earlier work this paper cites.
Online learning and online convex optimization
Shai Shalev-Shwartz et al · 2012
Earlier work this paper cites.
Cost-sensitive online active learning with application to malicious URL detection
Peilin Zhao and Steven CH Hoi · 2013
Earlier work this paper cites.
Clustering data streams using grid-based synopsis
Vasudha Bhatnagar, Sharanjit Kaur, and Sharma Chakravarthy · 2014
Earlier work this paper cites.
Batch normalization: Accelerating deep network training by reducing internal covariate shift
Sergey Ioffe and Christian Szegedy · 2015
Earlier work this paper cites.
Layer normalization
Original
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton · 2016
Earlier work this paper cites.
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2016
Earlier work this paper cites.
On kernelized multi-armed bandits
Sayak Ray Chowdhury and Aditya Gopalan · 2017
Earlier work this paper cites.
Stargan: Unified generative adversarial networks for multi-domain image-to-image translation
Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Ha, Sunghun Kim, and Jaegul Choo · 2018
Earlier work this paper cites.
Unsupervised representation learning by predicting image rotations
Spyros Gidaris, Praveer Singh, and Nikos Komodakis · 2018
Earlier work this paper cites.
Decoupled weight decay regularization
Ilya Loshchilov and Frank Hutter · 2018
Earlier work this paper cites.
Multi-adversarial domain adaptation
Zhongyi Pei, Zhangjie Cao, Mingsheng Long, and Jianmin Wang · 2018
Earlier work this paper cites.
Do cifar-10 classifiers generalize to cifar-10?
Original
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar · 2018
Earlier work this paper cites.
Maximum classifier discrepancy for unsupervised domain adaptation
Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada · 2018
Earlier work this paper cites.
Generalizing across domains via cross-gradient training
Shiv Shankar, Vihari Piratla, Soumen Chakrabarti, Siddhartha Chaudhuri, Preethi Jyothi, and Sunita Sarawagi · 2018
Earlier work this paper cites.
Non-local neural networks
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He · 2018
Earlier work this paper cites.
Group normalization
Yuxin Wu and Kaiming He · 2018
Earlier work this paper cites.
Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz · 2019
Earlier work this paper cites.
Domain generalization via model-agnostic learning of semantic features
Qi Dou, Daniel Coelho de Castro, Konstantinos Kamnitsas, and Ben Glocker · 2019
Earlier work this paper cites.

Similar

Benchmarking neural network robustness to common corruptions and perturbations
Dan Hendrycks and Thomas Dietterich · 2019
Cited alongside, same era.
Fast autoaugment
Sungbin Lim, Ildoo Kim, Taesup Kim, Chiheon Kim, and Sungwoong Kim · 2019
Cited alongside, same era.
Robustness properties of facebook’s resnext WSL models
Original
A Emin Orhan · 2019
Cited alongside, same era.
Pytorch image models
Ross Wightman · 2019
Cited alongside, same era.
Partially observable multi-sensor sequential change detection: A combinatorial multi-armed bandit approach
Chen Zhang and Steven CH Hoi · 2019
Cited alongside, same era.
Online adaptive asymmetric active learning with limited budgets
Yifan Zhang, Peilin Zhao, Shuaicheng Niu, Qingyao Wu, Jiezhang Cao, Junzhou Huang, and Mingkui Tan · 2019
Cited alongside, same era.
A simple framework for contrastive learning of visual representations
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton · 2020
Cited alongside, same era.
Augmix: A simple data processing method to improve robustness and uncertainty
Dan Hendrycks, Norman Mu, Ekin D Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan · 2020
Cited alongside, same era.
Universal source-free domain adaptation
Jogendra Nath Kundu, Naveen Venkat, R Venkatesh Babu, et al · 2020
Cited alongside, same era.
Model adaptation: Unsupervised domain adaptation without source data
Rui Li, Qianfen Jiao, Wenming Cao, Hau-San Wong, and Si Wu · 2020
Cited alongside, same era.
Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation
Jian Liang, Dapeng Hu, and Jiashi Feng · 2020
Cited alongside, same era.
Evaluating prediction-time batch normalization for robustness under covariate shift
Original
Zachary Nado, Shreyas Padhy, D Sculley, Alexander D’Amour, Balaji Lakshminarayanan, and Jasper Snoek · 2020
Cited alongside, same era.

Then

ASAM: adaptive sharpness-aware minimization for scale-invariant learning of deep neural networks
Jungmin Kwon, Jeongseop Kim, Hyunseo Park, and In Kwon Choi · 2021
Later among the works it cites.
On feature normalization and data augmentation
Boyi Li, Felix Wu, Ser-Nam Lim, Serge Belongie, and Kilian Q Weinberger · 2021
Later among the works it cites.
TTT++: when does self-supervised test-time training fail or thrive?
Yuejiang Liu, Parth Kothari, Bastien van Delft, Baptiste Bellot-Gurlet, Taylor Mordan, and Alexandre Alahi · 2021
Later among the works it cites.
Test-time adaptation to distribution shift by confidence maximization and input transformation
Original
Chaithanya Kumar Mummadi, Robin Hutmacher, Kilian Rambach, Evgeny Levinkov, Thomas Brox, and Jan Hendrik Metzen · 2021
Later among the works it cites.
Source-free domain adaptation via avatar prototype generation and adaptation
Zhen Qiu, Yifan Zhang, Hongbin Lin, Shuaicheng Niu, Yanxia Liu, Qing Du, and Mingkui Tan · 2021
Later among the works it cites.
Online unsupervised learning of visual representations and categories
Original
Mengye Ren, Tyler R Scott, Michael L Iuzzolino, Michael C Mozer, and Richard Zemel · 2021
Later among the works it cites.
Tent: Fully test-time adaptation by entropy minimization
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell · 2021
Later among the works it cites.
Rethinking ”batch” in batchnorm
Original
Yuxin Wu and Justin Johnson · 2021
Later among the works it cites.
Regularizing neural networks via adversarial model perturbation
Yaowei Zheng, Richong Zhang, and Yongyi Mao · 2021
Later among the works it cites.
MT3: meta test-time training for self-supervised test-time adaption
Alexander Bartler, Andre Bühler, Felix Wiewel, Mario Döbler, and Bin Yang · 2022
Later among the works it cites.
Visda-2021 competition: Universal domain adaptation to improve performance on out-of-distribution data
Dina Bashkirova, Dan Hendrycks, Donghyun Kim, Haojin Liao, Samarth Mishra, Chandramouli Rajagopalan, Kate Saenko, Kuniaki Saito, Burhan Ul Tayyab, Piotr Teterwak, et al · 2022
Later among the works it cites.
Improving test-time adaptation via shift-agnostic weight regularization and nearest source prototypes
Sungha Choi, Seunghan Yang, Seokeon Choi, and Sungrack Yun · 2022
Later among the works it cites.
Efficient sharpness-aware minimization for improved training of neural networks
Jiawei Du, Hanshu Yan, Jiashi Feng, Joey Tianyi Zhou, Liangli Zhen, Rick Siow Mong Goh, and Vincent Tan · 2022
Later among the works it cites.
Back to the source: Diffusion-driven test-time adaptation
Jin Gao, Jialing Zhang, Xihui Liu, Evan Shelhamer, Trevor Darrell, and Dequan Wang · 2022
Later among the works it cites.
Prototype-guided continual adaptation for class-incremental unsupervised domain adaptation
Hongbin Lin, Yifan Zhang, Zhen Qiu, Shuaicheng Niu, Chuang Gan, Yanxia Liu, and Mingkui Tan · 2022
Later among the works it cites.
A convnet for the 2020s
Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie · 2022
Later among the works it cites.
Continual test-time domain adaptation
Qin Wang, Olga Fink, Luc Van Gool, and Dengxin Dai · 2022
Later among the works it cites.
Improving out-of-distribution robustness via selective augmentation
Huaxiu Yao, Yu Wang, Sai Li, Linjun Zhang, Weixin Liang, James Zou, and Chelsea Finn · 2022
Later among the works it cites.
Memo: Test time robustness via adaptation and augmentation
Marvin Zhang, Sergey Levine, and Chelsea Finn · 2022
Later among the works it cites.
MECTA: Memory-economic continual test-time model adaptation
Junyuan Hong, Lingjuan Lyu, Jiayu Zhou, and Michael Spranger · 2023
Closest in time.
TTN: A domain-shift aware batch normalization in test-time adaptation
Hyesu Lim, Byeonggeun Kim, Jaegul Choo, and Sungha Choi · 2023
Closest in time.
DELTA: Degradation-free fully test-time adaptation
Bowen Zhao, Chen Chen, and Shu-Tao Xia · 2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…