Uncovering Adversarial Risks of Test-Time Adaptation

Understand

Recently, test-time adaptation (TTA) has been proposed as a promising solution for addressing distribution shifts.

• It allows a base model to adapt to an unforeseen distribution during inference by leveraging the information from the batch of (unlabeled) test data.
• However, we uncover a novel security vulnerability of TTA based on the insight that predictions on benign samples can be impacted by malicious samples in the same batch.
• To exploit this vulnerability, we propose Distribution Invading Attack (DIA), which injects a small fraction of malicious data into the test batch.