2020

Self-Challenging Improves Cross-Domain Generalization

Huang, Zeyi, Wang, Haohan, Xing, Eric P. et al.

Understand

Convolutional Neural Networks (CNN) conduct image classification by activating dominant features that correlated with labels.

When the training and testing data are under similar distributions, their dominant features are similar, which usually facilitates decent performance on the testing data.
The performance is nonetheless unmet when tested on samples from different distributions, leading to the challenges in cross-domain image classification.
We introduce a simple training heuristic, Representation Self-Challenging (RSC), that significantly improves the generalization of CNN to the out-of-domain data.

Built on

Bridle, J.S., Cox, S.J.: Recnorm: Simultaneous normalisation and classification applied to speech recognition. In: Advances in Neural Information Processing Systems. pp. 234–240 (1991)
1991
Earlier work this paper cites.
Nowlan, S.J., Hinton, G.E.: Simplifying neural networks by soft weight-sharing. Neural computation 4
1992
Earlier work this paper cites.
Mitchell, T.M., et al.: Machine learning. 1997. Burr Ridge, IL: McGraw Hill 45
1997
Earlier work this paper cites.
Ben-David, S., Blitzer, J., Crammer, K., Kulesza, A., Pereira, F., Vaughan, J.W.: A theory of learning from different domains. Machine learning 79
2010
Earlier work this paper cites.
Torralba, A., Efros, A.A., et al.: Unbiased look at dataset bias. In: CVPR. vol. 1, p. 7. Citeseer (2011)
2011
Earlier work this paper cites.
Muandet, K., Balduzzi, D., Schölkopf, B.: Domain generalization via invariant feature representation. In: International Conference on Machine Learning. pp. 10–18 (2013)
2013
Earlier work this paper cites.
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research 15
2014
Earlier work this paper cites.
Ghifary, M., Bastiaan Kleijn, W., Zhang, M., Balduzzi, D.: Domain generalization for object recognition with multi-task autoencoders. In: Proceedings of the IEEE international conference on computer vision. pp. 2551–2559 (2015)
2015
Earlier work this paper cites.
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., Fei-Fei, L.: ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV) 115
2015
Earlier work this paper cites.
Tompson, J., Goroshin, R., Jain, A., LeCun, Y., Bregler, C.: Efficient object localization using convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 648–656 (2015)
2015
Earlier work this paper cites.
Larsson, G., Maire, M., Shakhnarovich, G.: Fractalnet: Ultra-deep neural networks without residuals. arXiv preprint arXiv:1605.07648 (2016)
Original
2016
Earlier work this paper cites.

Similar

Park, S., Kwak, N.: Analysis on the dropout effect in convolutional neural networks. In: Asian conference on computer vision. pp. 189–204. Springer (2016)
2016
Cited alongside, same era.
Csurka, G.: Domain adaptation for visual applications: A comprehensive survey. arXiv preprint arXiv:1702.05374 (2017)
Original
2017
Cited alongside, same era.
DeVries, T., Taylor, G.W.: Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552 (2017)
Original
2017
Cited alongside, same era.
Gastaldi, X.: Shake-shake regularization. arXiv preprint arXiv:1705.07485 (2017)
Original
2017
Cited alongside, same era.
Li, D., Yang, Y., Song, Y.Z., Hospedales, T.M.: Deeper, broader and artier domain generalization. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 5542–5550 (2017)
2017
Cited alongside, same era.
Morerio, P., Cavazza, J., Volpi, R., Vidal, R., Murino, V.: Curriculum dropout. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 3544–3552 (2017)
2017
Cited alongside, same era.
Singh, K.K., Lee, Y.J.: Hide-and-seek: Forcing a network to be meticulous for weakly-supervised object and action localization. In: 2017 IEEE international conference on computer vision (ICCV). pp. 3544–3553. IEEE (2017)
2017
Cited alongside, same era.
Venkateswara, H., Eusebio, J., Chakraborty, S., Panchanathan, S.: Deep hashing network for unsupervised domain adaptation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 5018–5027 (2017)
2017
Cited alongside, same era.
Balaji, Y., Sankaranarayanan, S., Chellappa, R.: Metareg: Towards domain generalization using meta-regularization. In: Advances in Neural Information Processing Systems. pp. 998–1008 (2018)
2018
Cited alongside, same era.
Ghiasi, G., Lin, T.Y., Le, Q.V.: Dropblock: A regularization method for convolutional networks. In: Advances in Neural Information Processing Systems. pp. 10727–10737 (2018)
2018
Cited alongside, same era.
Li, H., Jialin Pan, S., Wang, S., Kot, A.C.: Domain generalization with adversarial feature learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 5400–5409 (2018)
2018
Cited alongside, same era.

Then

Park, S., Park, J., Shin, S.J., Moon, I.C.: Adversarial dropout for supervised and semi-supervised learning. In: Thirty-Second AAAI Conference on Artificial Intelligence (2018)
2018
Later among the works it cites.
Shankar, S., Piratla, V., Chakrabarti, S., Chaudhuri, S., Jyothi, P., Sarawagi, S.: Generalizing across domains via cross-gradient training. arXiv preprint arXiv:1804.10745 (2018)
Original
2018
Later among the works it cites.
Volpi, R., Namkoong, H., Sener, O., Duchi, J.C., Murino, V., Savarese, S.: Generalizing to unseen domains via adversarial data augmentation. In: Advances in Neural Information Processing Systems. pp. 5334–5344 (2018)
2018
Later among the works it cites.
Wang, M., Deng, W.: Deep visual domain adaptation: A survey. Neurocomputing 312
2018
Later among the works it cites.
Carlucci, F.M., D’Innocente, A., Bucci, S., Caputo, B., Tommasi, T.: Domain generalization by solving jigsaw puzzles. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 2229–2238 (2019)
2019
Later among the works it cites.
Dou, Q., Castro, D.C., Kamnitsas, K., Glocker, B.: Domain generalization via model-agnostic learning of semantic features. arXiv preprint arXiv:1910.13580 (2019)
Original
2019
Later among the works it cites.
Lee, S., Kim, D., Kim, N., Jeong, S.G.: Drop to adapt: Learning discriminative features for unsupervised domain adaptation. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 91–100 (2019)
2019
Later among the works it cites.
Li, D., Zhang, J., Yang, Y., Liu, C., Song, Y.Z., Hospedales, T.M.: Episodic training for domain generalization. arXiv preprint arXiv:1902.00113 (2019)
Original
2019
Later among the works it cites.
Wang, H., Ge, S., Xing, E.P., Lipton, Z.C.: Learning robust global representations by penalizing local predictive power. In: Advances in Neural Information Processing Systems (NeurIPS 2019) (2019)
2019
Later among the works it cites.
Wang, H., He, Z., Lipton, Z.C., Xing, E.P.: Learning robust representations by projecting superficial statistics out. In: International Conference on Learning Representations (2019)
2019
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…