Identifying the Context Shift between Test Benchmarks and Production Data

M. Arjovsky, L. Bottou, I. Gulrajani, and D. Lopez-Paz, “Invariant Risk Minimization,” Mar. 2020, number: arXiv:1907.02893 arXiv:1907.02893 [cs, stat]. [Online]. Available: http://arxiv.org/abs/1907.02893

Original

1907

Earlier work this paper cites.

B. Dolhansky, R. Howes, B. Pflaum, N. Baram, and C. C. Ferrer, “The Deepfake Detection Challenge (DFDC) Preview Dataset,” arXiv:1910.08854 [cs] , Oct. 2019, arXiv: 1910.08854. [Online]. Available: http://arxiv.org/abs/1910.08854

Original

1910

Earlier work this paper cites.

S. Kerr, “On the folly of rewarding a, while hoping for b,” Academy of Management journal , vol. 18, no. 4, pp. 769–783, 1975

1975

Earlier work this paper cites.

D. E. Forsythe, “Engineering knowledge: The construction of knowledge in artificial intelligence,” Social studies of science , vol. 23, no. 3, pp. 445–477, 1993

1993

Earlier work this paper cites.

W. Ickes, “Empathic accuracy,” Journal of personality , vol. 61, no. 4, pp. 587–610, 1993

1993

Earlier work this paper cites.

G. Widmer and M. Kubat, “Learning in the presence of concept drift and hidden contexts,” Machine learning , vol. 23, no. 1, pp. 69–101, 1996

1996

Earlier work this paper cites.

R. W. Picard, Affective computing , 2000

2000

Earlier work this paper cites.

R. Kosti, J. M. Alvarez, A. Recasens, and A. Lapedriza, “Context Based Emotion Recognition using EMOTIC Dataset,” IEEE Transactions on Pattern Analysis and Machine Intelligence , pp. 1–1, 2019, arXiv: 2003.13401. [Online]. Available: http://arxiv.org/abs/2003.13401

Original

2003

Earlier work this paper cites.

S. Lyu, “DeepFake Detection: Current Challenges and Next Steps,” arXiv:2003.09234 [cs] , Mar. 2020, arXiv: 2003.09234. [Online]. Available: http://arxiv.org/abs/2003.09234

Original

2003

Earlier work this paper cites.

M. Pantic, M. Valstar, R. Rademaker, and L. Maat, “Web-based database for facial expression analysis,” in 2005 IEEE international conference on multimedia and Expo . IEEE, 2005, pp. 5–pp

2005

Earlier work this paper cites.

B. Dolhansky, J. Bitton, B. Pflaum, J. Lu, R. Howes, M. Wang, and C. C. Ferrer, “The DeepFake Detection Challenge (DFDC) Dataset,” arXiv:2006.07397 [cs] , Oct. 2020, arXiv: 2006.07397. [Online]. Available: http://arxiv.org/abs/2006.07397

Original

2006

Earlier work this paper cites.

J. Quiñonero-Candela, M. Sugiyama, A. Schwaighofer, and N. D. Lawrence, Dataset shift in machine learning , 2008

2008

Earlier work this paper cites.

S. Santurkar, D. Tsipras, and A. Madry, “BREEDS: Benchmarks for Subpopulation Shift,” Aug. 2020, number: arXiv:2008.04859 arXiv:2008.04859 [cs, stat]. [Online]. Available: http://arxiv.org/abs/2008.04859

Original

2008

Earlier work this paper cites.

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in 2009 IEEE conference on computer vision and pattern recognition . Ieee, 2009, pp. 248–255

2009

Earlier work this paper cites.

I. Y. Chen, E. Pierson, S. Rose, S. Joshi, K. Ferryman, and M. Ghassemi, “Ethical Machine Learning in Health Care,” arXiv:2009.10576 [cs] , Oct. 2020, arXiv: 2009.10576. [Online]. Available: http://arxiv.org/abs/2009.10576

Original

2009

Earlier work this paper cites.

T. Bänziger and K. R. Scherer, “Introducing the geneva multimodal emotion portrayal (gemep) corpus,” Blueprint for affective computing: A sourcebook , vol. 2010, pp. 271–94, 2010

2010

Earlier work this paper cites.

P. Lucey, J. F. Cohn, T. Kanade, J. Saragih, Z. Ambadar, and I. Matthews, “The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression,” in 2010 ieee computer society conference on computer vision and pattern recognition-workshops . IEEE, 2010, pp. 94–101

2010

Earlier work this paper cites.

R. Gross, I. Matthews, J. Cohn, T. Kanade, and S. Baker, “Multi-pie,” Image and vision computing , vol. 28, no. 5, pp. 807–813, 2010

2010

Earlier work this paper cites.

A. Torralba and A. A. Efros, “Unbiased look at dataset bias,” in CVPR 2011 . IEEE, 2011, pp. 1521–1528

2011

Earlier work this paper cites.

A. Dhall, R. Goecke, S. Lucey, and T. Gedeon, “Static facial expression analysis in tough conditions: Data, evaluation protocol and benchmark,” in 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops) . IEEE, 2011, pp. 2106–2112

2011

Earlier work this paper cites.

M. F. Valstar, B. Jiang, M. Mehu, M. Pantic, and K. Scherer, “The first facial expression recognition and analysis challenge,” in 2011 IEEE International Conference on Automatic Face & Gesture Recognition (FG) . IEEE, 2011, pp. 921–926

2011

Earlier work this paper cites.

L. Deng, “The mnist database of handwritten digit images for machine learning research [best of the web],” IEEE signal processing magazine , vol. 29, no. 6, pp. 141–142, 2012

2012

Earlier work this paper cites.

J. G. Moreno-Torres, T. Raeder, R. Alaiz-Rodríguez, N. V. Chawla, and F. Herrera, “A unifying view on dataset shift in classification,” Pattern recognition , vol. 45, no. 1, pp. 521–530, 2012

2012

Earlier work this paper cites.

P. W. Koh, S. Sagawa, H. Marklund, S. M. Xie, M. Zhang, A. Balsubramani, W. Hu, M. Yasunaga, R. L. Phillips, I. Gao, T. Lee, E. David, I. Stavness, W. Guo, B. A. Earnshaw, I. S. Haque, S. Beery, J. Leskovec, A. Kundaje, E. Pierson, S. Levine, C. Finn, and P. Liang, “WILDS: A Benchmark of in-the-Wild Distribution Shifts,” arXiv:2012.07421 [cs] , Jul. 2021, arXiv: 2012.07421. [Online]. Available: http://arxiv.org/abs/2012.07421

Original

2012

Earlier work this paper cites.

A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” Advances in neural information processing systems , vol. 25, 2012

2012

Earlier work this paper cites.

S. M. Mavadati, M. H. Mahoor, K. Bartlett, P. Trinh, and J. F. Cohn, “Disfa: A spontaneous facial action intensity database,” IEEE Transactions on Affective Computing , vol. 4, no. 2, pp. 151–160, 2013

2013

Earlier work this paper cites.

I. J. Goodfellow, D. Erhan, P. L. Carrier, A. Courville, M. Mirza, B. Hamner, W. Cukierski, Y. Tang, D. Thaler, D.-H. Lee et al. , “Challenges in representation learning: A report on three machine learning contests,” in International conference on neural information processing . Springer, 2013, pp. 117–124

2013

Earlier work this paper cites.

A. Krizhevsky, V. Nair, and G. Hinton, “The cifar-10 dataset,” online: http://www. cs. toronto. edu/kriz/cifar. html , vol. 55, no. 5, 2014

2014

Earlier work this paper cites.

X. Chen, H. Fang, T.-Y. Lin, R. Vedantam, S. Gupta, P. Dollár, and C. L. Zitnick, “Microsoft coco captions: Data collection and evaluation server,” arXiv preprint arXiv:1504.00325 , 2015

Original

2015

Earlier work this paper cites.

O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein et al. , “Imagenet large scale visual recognition challenge,” International journal of computer vision , vol. 115, no. 3, pp. 211–252, 2015

2015

Earlier work this paper cites.

J. F. Cohn and F. De la Torre, “Automated face analysis for affective computing.” 2015

2015

Earlier work this paper cites.

A. E. Johnson, T. J. Pollard, L. Shen, L.-w. H. Lehman, M. Feng, M. Ghassemi, B. Moody, P. Szolovits, L. Anthony Celi, and R. G. Mark, “Mimic-iii, a freely accessible critical care database,” Scientific data , vol. 3, no. 1, pp. 1–9, 2016

2016

Earlier work this paper cites.

A. Mollahosseini, D. Chan, and M. H. Mahoor, “Going deeper in facial expression recognition using deep neural networks,” in 2016 IEEE Winter conference on applications of computer vision (WACV) . IEEE, 2016, pp. 1–10

2016

Earlier work this paper cites.

X. Chen, C. Liu, B. Li, K. Lu, and D. Song, “Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning,” arXiv:1712.05526 [cs] , Dec. 2017, arXiv: 1712.05526. [Online]. Available: http://arxiv.org/abs/1712.05526

Original

2017

Earlier work this paper cites.

T. B. Brown, D. Mané, A. Roy, M. Abadi, and J. Gilmer, “Adversarial patch,” arXiv preprint arXiv:1712.09665 , 2017

Original

2017

Earlier work this paper cites.

S. Mullainathan and Z. Obermeyer, “Does Machine Learning Automate Moral Hazard and Error?” vol. 107, no. 5, p. 5, 2017

2017

Cited alongside, same era.

A. Paiva, I. Leite, H. Boukricha, and I. Wachsmuth, “Empathy in virtual agents and robots: A survey,” ACM Transactions on Interactive Intelligent Systems (TiiS) , vol. 7, no. 3, pp. 1–40, 2017

2017

Cited alongside, same era.

A. Esteva, B. Kuprel, R. A. Novoa, J. Ko, S. M. Swetter, H. M. Blau, and S. Thrun, “Dermatologist-level classification of skin cancer with deep neural networks,” Nature , vol. 542, no. 7639, pp. 115–118, Feb. 2017. [Online]. Available: http://www.nature.com/articles/nature21056

2017

Cited alongside, same era.

A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman, “Glue: A multi-task benchmark and analysis platform for natural language understanding,” arXiv preprint arXiv:1804.07461 , 2018

Original

2018

Cited alongside, same era.

A. Ghandeharioun, B. Kim, C.-L. Li, B. Jou, B. Eoff, and R. W. Picard, “Dissect: Disentangled simultaneous explanations via concept traversals,” arXiv preprint arXiv:2105.15164 , 2021

Original

2021

Later among the works it cites.

M. Groh, C. Harris, L. Soenksen, F. Lau, R. Han, A. Kim, A. Koochek, and O. Badri, “Evaluating Deep Neural Networks Trained on Clinical Images in Dermatology with the Fitzpatrick 17k Dataset,” in 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) . Nashville, TN, USA: IEEE, Jun. 2021, pp. 1820–1828. [Online]. Available: https://ieeexplore.ieee.org/document/9522867/

2021

Later among the works it cites.

B. Schölkopf, F. Locatello, S. Bauer, N. R. Ke, N. Kalchbrenner, A. Goyal, and Y. Bengio, “Toward causal representation learning,” Proceedings of the IEEE , vol. 109, no. 5, pp. 612–634, 2021

2021

Later among the works it cites.

T. Gebru, J. Morgenstern, B. Vecchione, J. W. Vaughan, H. Wallach, H. D. Iii, and K. Crawford, “Datasheets for datasets,” Communications of the ACM , vol. 64, no. 12, pp. 86–92, 2021