Benchmarks for Automated Commonsense Reasoning: A Survey

Thomee, Bart, David A. Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. “YFCC100M: The new data in multimedia research.” Communications of the ACM 59, no. 2 (2016): 64-73. https://dl.acm.org/doi/abs/10.1145/2812802

2016

Cited alongside, same era.

Davis, Ernest. “The Logical Depth of Reasoning about Other Minds.” Advances in Cognitive Systems,

2017

Cited alongside, same era.

Evans, Jonathan St BT. “Dual process theory: Perspectives and problems.” In Wim de Neys (ed.) Dual process theory 2.0

2017

Cited alongside, same era.

Gordon, Andrew S., and Jerry R. Hobbs. A formal theory of commonsense psychology: How people think people think

2017

Cited alongside, same era.

Goyal, Raghav, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel et al. “The ‘something something’ video database for learning and evaluating visual common sense.” In Proceedings of the IEEE international conference on computer vision, pp. 5842-5850. 2017. https://openaccess.thecvf.com/content_ICCV_2017/papers/Goyal_The_Something_Something_ICCV_2017_paper.pdf

2017

Cited alongside, same era.

Krishna, Ranjay, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen et al. “Visual genome: Connecting language and vision using crowdsourced dense image annotations.” International journal of computer vision 123, no. 1 (2017): 32-73. https://link.springer.com/article/10.1007/S11263-016-0981-7

2017

Cited alongside, same era.

Rohrbach, Anna, Atousa Torabi, Marcus Rohrbach, Niket Tandon, Christopher Pal, Hugo Larochelle, Aaron Courville, and Bernt Schiele. “Movie description.” International Journal of Computer Vision 123, no. 1 (2017): 94-120. https://link.springer.com/article/10.1007/s11263-016-0987-1

2017

Cited alongside, same era.

Speer, Robyn, Joshua Chin, and Catherine Havasi. “ConceptNet 5.5: An open multilingual graph of general knowledge.” In Thirty-first AAAI conference on artificial intelligence. 2017. https://ojs.aaai.org/index.php/AAAI/article/view/11164

2017

Cited alongside, same era.

Lal, Yash Kumar, Nathanael Chambers, Raymond Mooney, and Niranjan Balasubramanian. “Tellmewhy: A dataset for answering why-questions in narratives.” arXiv preprint arXiv:2106.06132 (2021). https://aclanthology.org/2021.findings-acl.53/

Original

2021

Later among the works it cites.

Li, Linjie, Jie Lei, Zhe Gan, and Jingjing Liu. “Adversarial VQA: A new benchmark for evaluating the robustness of vqa models.” In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2042-2051. 2021. https://openaccess.thecvf.com/content/ICCV2021/html/Li_Adversarial_VQA_A_New_Benchmark_for_Evaluating_the_Robustness_of_ICCV_2021_paper.html

2021

Later among the works it cites.

Lin, Bill Yuchen, Seyeon Lee, Xiaoyang Qiao, and Xiang Ren. “Common sense beyond English: Evaluating and improving multilingual language models for commonsense reasoning.” arXiv preprint arXiv:2106.06937 (2021). https://arxiv.org/abs/2106.06937

Original

2021

Later among the works it cites.

Lin, Bill Yuchen, Ziyi Wu, Yichi Yang, Dong-Ho Lee, and Xiang Ren. “RiddleSense: Reasoning about riddle questions featuring linguistic creativity and commonsense knowledge.” arXiv preprint arXiv:2101.00376 (2021). https://arxiv.org/abs/2101.00376

Original

2021

Later among the works it cites.

Lu, Pan, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chun Zhu. “IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning.” arXiv preprint arXiv:2110.13214 (2021). https://arxiv.org/abs/2110.13214

Original

2021

Later among the works it cites.

Onoe, Yasumasa, Michael JQ Zhang, Eunsol Choi, and Greg Durrett. “CREAK: A dataset for commonsense reasoning over entity knowledge.” arXiv preprint arXiv:2109.01653 (2021). https://arxiv.org/abs/2109.01653

Original

2021

Later among the works it cites.

Parrish, Alicia, Sebastian Schuster, Alex Warstadt, Omar Agha, Soo-Hwan Lee, Zhuoye Zhao, Samuel R. Bowman, and Tal Linzen. “NOPE: A Corpus of Naturally-Occurring Presuppositions in English.“ arXiv preprint arXiv:2109.06987 (2021). https://aclanthology.org/2021.conll-1.28/

Original

2021

Later among the works it cites.

Patel, Arkil, Satwik Bhattamishra, and Navin Goyal. “Are NLP Models really able to Solve Simple Math Word Problems?.” arXiv preprint arXiv:2103.07191 (2021). https://arxiv.org/abs/2103.07191

Original

2021

Later among the works it cites.

Paullada, Amandalynne, Inioluwa Deborah Raji, Emily M. Bender, Emily Denton, and Alex Hanna. “Data and its (dis) contents: A survey of dataset development and use in machine learning research.” Patterns 2, no. 11 (2021): 100336. https://www.sciencedirect.com/science/article/pii/S2666389921001847

2021

Later among the works it cites.

Qin, Lianhui, Aditya Gupta, Shyam Upadhyay, Luheng He, Yejin Choi, and Manaal Faruqui. “TIMEDIAL: Temporal commonsense reasoning in dialog.” arXiv preprint arXiv:2106.04571 (2021). https://arxiv.org/abs/2106.04571

Original

2021

Later among the works it cites.

Riochet, Ronan, Mario Ynocente Castro, Mathieu Bernard, Adam Lerer, Rob Fergus, Véronique Izard, and Emmanuel Dupoux. “IntPhys 2019: A Benchmark for Visual Intuitive Physics Understanding.” IEEE Transactions on Pattern Analysis and Machine Intelligence 44, no. 9 (2021): 5016-5025. https://ieeexplore.ieee.org/abstract/document/9442261

2021

Later among the works it cites.

Sakaguchi, Keisuke, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. “Winogrande: An adversarial Winograd schema challenge at scale.” Communications of the ACM 64, no. 9 (2021): 99-106. https://dl.acm.org/doi/abs/10.1145/3474381

2021

Later among the works it cites.

Seo, Jaehyung, Chanjun Park, Hyeonseok Moon, Sugyeong Eo, Myunghoon Kang, Seounghoon Lee, and Heuiseok Lim. “KommonGen: A Dataset for Korean Generative Commonsense Reasoning Evaluation.” In Annual Conference on Human and Language Technology, pp. 55-60. Human and Language Technology, 2021. https://koreascience.kr/article/CFKO202130060697830.pdf

2021

Later among the works it cites.

Shu, Tianmin, Abhishek Bhandwaldar, Chuang Gan, Kevin Smith, Shari Liu, Dan Gutfreund, Elizabeth Spelke, Joshua Tenenbaum, and Tomer Ullman. “Agent: A benchmark for core psychological reasoning.” In International Conference on Machine Learning, pp. 9614-9625. PMLR, 2021. https://research.ibm.com/publications/agent-a-benchmark-for-core-psychological-reasoning

2021

Later among the works it cites.

Singh, Shikhar, Nuan Wen, Yu Hou, Pegah Alipoormolabashi, Te-Lin Wu, Xuezhe Ma, and Nanyun Peng. “COM2SENSE: A commonsense reasoning benchmark with complementary sentences.” arXiv preprint arXiv:2106.00969 (2021). https://arxiv.org/abs/2106.00969

Original

2021

Later among the works it cites.

KVL-BERT: Knowledge Enhanced Visual-and-Linguistic BERT for visual commonsense reasoning,

Song, Dandan, Siyi Ma, Zhanchen Sun, Sicheng Yang, Lejian Lia (2021) · 2021

Later among the works it cites.

Shane Storks, Qiaozi Gao, Yichi Zhang, Joyce Chai. “Tiered Reasoning for Intuitive Physics: Toward Verifiable Commonsense Language Understanding.” arXiv preprint arXiv:2109.04947 (2021). https://arxiv.org/abs/2109.04947

Original

2021

Later among the works it cites.

Weihs, Luca, Matt Deitke, Aniruddha Kembhavi, and Roozbeh Mottaghi. “Visual room rearrangement.” In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5922-5931. 2021. https://openaccess.thecvf.com/content/CVPR2021/html/Weihs_Visual_Room_Rearrangement_CVPR_2021_paper.html

2021

Later among the works it cites.

Yin, Da, Liunian Harold Li, Ziniu Hu, Nanyun Peng, and Kai-Wei Chang. “Broaden the Vision: Geo-Diverse Visual Commonsense Reasoning.” arXiv preprint arXiv:2109.06860 (2021). https://arxiv.org/abs/2109.06860

Original

2021

Later among the works it cites.

Aghahadi, Zeinab, and Alireza Talebpour. “Avicenna: a challenge dataset for natural language generation toward commonsense syllogistic reasoning.” Journal of Applied Non-Classical Logics

2022

Later among the works it cites.

Ashida, Mana, and Saku Sugawara. “Possible Stories: Evaluating Situated Commonsense Reasoning under Multiple Possible Scenarios.” arXiv preprint arXiv:2209.07760 (2022). https://arxiv.org/abs/2209.07760

Original

2022

Later among the works it cites.

Bhargava, Prajjwal, and Vincent Ng. “Commonsense Knowledge Reasoning and Generation with Pre-trained Language Models: A Survey.” arXiv preprint arXiv:2201.12438 (2022). https://www.aaai.org/AAAI22Papers/SMT-00456-BhargavaP.pdf

Original

2022

Later among the works it cites.

Bitton, Yonatan, Nitzan Bitton Guetta, Ron Yosef, Yuval Elovici, Mohit Bansal, Gabriel Stanovsky, and Roy Schwartz. “Winogavil: Gamified association benchmark to challenge vision-and-language models.” arXiv preprint arXiv:2207.12576 (2022). https://arxiv.org/abs/2207.12576

Original

2022

Later among the works it cites.

Over-reliance on English hinders cognitive science

Blasi, Damián, Joseph Hencirh, Evangelia Adamou, David Kemmerer, Asifa Majid (2022) · 2022

Later among the works it cites.

Davis, Ernest and Gary Marcus. “Experiments in Commonsense Reasoning in GPT-3: Status Report from June 2022.” Unpublished. https://cs.nyu.edu/~davise/papers/GPT-3-6-22.html

2022

Later among the works it cites.

Ding, Yan, Xiaohan Zhang, Saeid Amiri, Nieqing Cao, Hao Yang, Chad Esselink, and Shiqi Zhang. “Robot Task Planning and Situation Handling in Open Worlds.” arXiv preprint arXiv:2210.01287 (2022). https://arxiv.org/abs/2210.01287

Original

2022

Later among the works it cites.

Gatti, Prajwal, Abhirama Subramanyam Penamakuri, Revant Teotia, Anand Mishra, Shubhashis Sengupta, and Roshni Ramnani. “COFAR: Commonsense and Factual Reasoning in Image Search.” arXiv preprint arXiv:2210.08554 (2022). https://arxiv.org/abs/2210.08554

Original

2022

Later among the works it cites.

Guan, Jian, Zhuoer Feng, Yamei Chen, Ruilin He, Xiaoxi Mao, Changjie Fan, and Minlie Huang. “LOT: A Story-Centric Benchmark for Evaluating Chinese Long Text Understanding and Generation.” Transactions of the Association for Computational Linguistics 10 (2022): 434-451

2022

Later among the works it cites.

Gupta, Himanshu, Neeraj Varshney, Swaroop Mishra, Kuntal Kumar Pal, Saurabh Arjun Sawant, Kevin Scaria, Siddharth Goyal, and Chitta Baral. “ ‘John is 50 years old, can his son be 65?’ Evaluating NLP Models’ Understanding of Feasibility.” arXiv preprint arXiv:2210.07471 (2022). https://arxiv.org/abs/2210.07471

Original

2022

Later among the works it cites.

Hessel, Jack, Jena D. Hwang, Jae Sung Park, Rowan Zellers, Chandra Bhagavatula, Anna Rohrbach, Kate Saenko, and Yejin Choi. “The abduction of Sherlock Holmes: A dataset for visual abductive reasoning.” arXiv preprint arXiv:2202.04800 (2022). https://arxiv.org/abs/2202.04800

Original

2022

Later among the works it cites.

Kant, Yash, Arun Ramachandran, Sriram Yenamandra, Igor Gilitschenski, Dhruv Batra, Andrew Szot, and Harsh Agrawal. “Housekeep: Tidying Virtual Households using Commonsense Reasoning.” arXiv preprint arXiv:2205.10712 (2022). https://arxiv.org/abs/2205.10712

Original

2022

Later among the works it cites.

Kim, Hyounghun, Abhay Zala, and Mohit Bansal. “CoSIm: commonsense reasoning for counterfactual scene imagination.” arXiv preprint arXiv:2207.03961 (2022). https://arxiv.org/abs/2207.03961

Original

2022

Later among the works it cites.

Kocijan, Vid, Ernest Davis, Thomas Lukasiewicz, Gary Marcus, and Leora Morgenstern. “The Defeat of the Winograd Schema Challenge.” arXiv preprint arXiv:2201.02387 (2022). https://arxiv.org/abs/2201.02387

Original

2022

Later among the works it cites.

Liu, Xiao, Da Yin, Yansong Feng, and Dongyan Zhao. “Things not written in text: Exploring spatial commonsense from visual signals.” arXiv preprint arXiv:2203.08075 (2022). https://arxiv.org/abs/2203.08075

Original

2022

Later among the works it cites.

Mishra, Swaroop, Matthew Finlayson, Pan Lu, Leonard Tang, Sean Welleck, Chitta Baral, Tanmay Rajpurohit et al. “ Līla

Original

2022

Later among the works it cites.

Santos, Henrique, Ke Shen, Alice M. Mulvehill, Yasaman Razeghi, Deborah L. McGuinness, and Mayank Kejriwal. “A Theoretically Grounded Benchmark for Evaluating Machine Commonsense.” arXiv preprint arXiv:2203.12184 (2022). https://arxiv.org/abs/2203.12184

Original

2022

Later among the works it cites.

Beyond the imitation game: Quantifying and extrapolating the capacities of language models

Original

Srivastaa, Aarohi et al. 2022 · 2022

Later among the works it cites.

Talmor, Alon, Ori Yoran, Ronan Le Bras, Chandra Bhagavatula, Yoav Goldberg, Yejin Choi, and Jonathan Berant. “Commonsenseqa 2.0: Exposing the limits of ai through gamification.” arXiv preprint arXiv:2201.05320 (2022). https://arxiv.org/abs/2201.05320

Original

2022

Later among the works it cites.

Not another Negation Benchmark: The NaN-NLI Test Suite for Sub-clausal Negation

Original

Truong, Hung Thinh, Yulia Otmakhova, Timothy Baldwin, Trevor Cohn, Karin Verspoor, and Jey Han Lau. (2022) · 2022

Later among the works it cites.

Wang, Ruoyao, Peter Jansen, Marc-Alexandre Côté, and Prithviraj Ammanabrolu. “ScienceWorld: Is your Agent Smarter than a 5th Grader?.” arXiv preprint arXiv:2203.07540 (2022). https://arxiv.org/abs/2203.07540

Original

2022

Later among the works it cites.

Weihs, Luca, Amanda Rose Yuile, Renée Baillargeon, Cynthia L. Fisher, Gary Marcus, Roozbeh Mottaghi, and Aniruddha Kembhavi. “Benchmarking progress to infant-Level physical reasoning in AI.” Transactions on Machine Learning, 2022. https://openreview.net/pdf?id=9NjqD9i48M

2022

Later among the works it cites.

Yin, Da, Hritik Bansal, Masoud Monajatipoor, Liunian Harold Li, and Kai-Wei Chang. “GeoMLAMA: Geo-Diverse Commonsense Probing on Multilingual Pre-Trained Language Models.” arXiv preprint arXiv:2205.12247 (2022). https://arxiv.org/abs/2205.12247

Original

2022

Later among the works it cites.

You, Haoxuan, Rui Sun, Zhecan Wang, Kai-Wei Chang, and Shih-Fu Chang. “Find Someone Who: Visual Commonsense Understanding in Human-Centric Grounding.” arXiv preprint arXiv:2212.06971 (2022). https://arxiv.org/abs/2212.06971

Original

2022

Later among the works it cites.

Yu, Samuel, Peter Wu, Paul Pu Liang, Ruslan Salakhutdinov, and Louis-Philippe Morency. “PACS: A Dataset for Physical Audiovisual CommonSense Reasoning.” In European Conference on Computer Vision, pp. 292-309. Springer, Cham, 2022. https://ui.adsabs.harvard.edu/abs/2022arXiv220311130Y/abstract

2022

Later among the works it cites.

Garcia, Xavier, Yamini Bansal, Colin Cherry, George Foster, Maxim Krikun, Fangxiaoyu Feng, Melvin Johnson, and Orhan Firat. ”The unreasonable effectiveness of few-shot learning for machine translation.” arXiv preprint arXiv:2302.01398 (2023) https://arxiv.org/abs/2302.01398

Original

2023

Closest in time.

Kosinski, Michael. “Theory of Mind may have Spontaneoulsy Emerged in Large Language Models”. (2023) arXiv preprint arXiv:2302.02083 https://arxiv.org/abs/2302.02083

Original

2023

Closest in time.

Marcus, Gary and Ernest Davis. “How Not to Test GPT.” Comm. ACM blog

2023

Closest in time.

LoBue, Peter, and Alexander Yates. “Types of common-sense knowledge needed for recognizing textual entailment.” In Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies, pp. 329-334. 2011. https://aclanthology.org/P11-2057.pdf

2057

Closest in time.