2023

GeomVerse: A Systematic Evaluation of Large Models for Geometric Reasoning

Kazemi, Mehran, Alvari, Hamidreza, Anand, Ankit et al.

Understand

Large language models have shown impressive results for multi-hop mathematical reasoning when the input question is only textual.

Many mathematical reasoning problems, however, contain both text and image.
With the ever-increasing adoption of vision language models (VLMs), understanding their reasoning abilities for such problems is crucial.
In this paper, we evaluate the reasoning capabilities of VLMs along various axes through the lens of geometry problems.

Built on

A machine-oriented logic based on the resolution principle
J. A. Robinson. 1965 · 1965
Earlier work this paper cites.
E–a brainiac theorem prover
Stephan Schulz. 2002 · 2002
Earlier work this paper cites.
First-order theorem proving and vampire
Laura Kovács and Andrei Voronkov. 2013 · 2013
Earlier work this paper cites.
Solving geometry problems: Combining text and diagram interpretation
Minjoon Seo, Hannaneh Hajishirzi, Ali Farhadi, Oren Etzioni, and Clint Malcolm. 2015 · 2015
Earlier work this paper cites.
Program induction by rationale generation: Learning to solve and explain algebraic word problems
Original
Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017 · 2017
Earlier work this paper cites.
Plotqa: Reasoning over scientific plots
Nitesh Methani, Pritha Ganguly, Mitesh M Khapra, and Pratyush Kumar. 2020 · 2020
Earlier work this paper cites.
IMO Grand Challenge
Daniel Selsam, Leonardo de Moura, Kevin Buzzard, Reid Barton, Percy Liang, Sarah Loos, and Freek Wiedijk. 2020 · 2020
Earlier work this paper cites.
Critical thinking for language models
Gregor Betz, Christian Voigt, and Kyle Richardson. 2021 · 2021
Earlier work this paper cites.
Geoqa: A geometric question answering benchmark towards multimodal numerical reasoning
Original
Jiaqi Chen, Jianheng Tang, Jinghui Qin, Xiaodan Liang, Lingbo Liu, Eric P Xing, and Liang Lin. 2021 · 2021
Earlier work this paper cites.
Transformers as soft reasoners over language
Peter Clark, Oyvind Tafjord, and Kyle Richardson. 2021 · 2021
Earlier work this paper cites.
Training verifiers to solve math word problems
Original
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021 · 2021
Earlier work this paper cites.
Explaining answers with entailment trees
Bhavana Dalvi, Peter Jansen, Oyvind Tafjord, Zhengnan Xie, Hannah Smith, Leighanna Pipatanangkura, and Peter Clark. 2021 · 2021
Earlier work this paper cites.
Advancing mathematics by guiding human intuition with AI
Alex Davies, Petar Veličković, Lars Buesing, Sam Blackwell, Daniel Zheng, Nenad Tomašev, Richard Tanburn, Peter Battaglia, Charles Blundell, András Juhász, Marc Lackenby, Geordie Williamson, Demis Hassabis, and Pushmeet Kohli. 2021 · 2021
Earlier work this paper cites.
Measuring mathematical problem solving with the math dataset
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021 · 2021
Earlier work this paper cites.
Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning
Original
Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chun Zhu. 2021 · 2021
Earlier work this paper cites.
ProofWriter: Generating implications, proofs, and abductive statements over natural language
Oyvind Tafjord, Bhavana Dalvi, and Peter Clark. 2021 · 2021
Earlier work this paper cites.

Similar

AR-LSAT: Investigating analytical reasoning of text
Original
Wanjun Zhong, Siyuan Wang, Duyu Tang, Zenan Xu, Daya Guo, Jiahai Wang, Jian Yin, Ming Zhou, and Nan Duan. 2021 · 2021
Cited alongside, same era.
Flamingo: a visual language model for few-shot learning
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. 2022 · 2022
Cited alongside, same era.
Penguins don’t fly: Reasoning about generics through instantiations and exceptions
Original
Emily Allaway, Jena D Hwang, Chandra Bhagavatula, Kathleen McKeown, Doug Downey, and Yejin Choi. 2022 · 2022
Cited alongside, same era.
Discovering faster matrix multiplication algorithms with reinforcement learning
Alhussein Fawzi, Matej Balog, Aja Huang, Thomas Hubert, Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Francisco J. R. Ruiz, Julian Schrittwieser, Grzegorz Swirszcz, David Silver, Demis Hassabis, and Pushmeet Kohli. 2022 · 2022
Cited alongside, same era.
Large language models can self-improve
Original
Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. 2022 · 2022
Cited alongside, same era.
Large language models are zero-shot reasoners
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022 · 2022
Cited alongside, same era.
Solving quantitative reasoning problems with language models
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. 2022 · 2022
Cited alongside, same era.
Clevr-math: A dataset for compositional language, visual and mathematical reasoning
Original
Adam Dahlgren Lindström and Savitha Sam Abraham. 2022 · 2022
Cited alongside, same era.
A survey of deep learning for mathematical reasoning
Original
Pan Lu, Liang Qiu, Wenhao Yu, Sean Welleck, and Kai-Wei Chang. 2022 · 2022
Cited alongside, same era.
Teaching small language models to reason
Original
Lucie Charlotte Magister, Jonathan Mallinson, Jakub Adamek, Eric Malmi, and Aliaksei Severyn. 2022 · 2022
Cited alongside, same era.
Chartqa: A benchmark for question answering about charts with visual and logical reasoning
Original
Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. 2022 · 2022
Cited alongside, same era.
Show your work: Scratchpads for intermediate computation with language models
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. 2022 · 2022
Cited alongside, same era.

Then

Neural story planning
Original
Anbang Ye, Christopher Cui, Taiwei Shi, and Mark O Riedl. 2022 · 2022
Later among the works it cites.
STaR: Bootstrapping reasoning with reasoning
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. 2022 · 2022
Later among the works it cites.
Gpt-3-driven pedagogical agents to train children’s curious question-asking skills
Rania Abdelghani, Yen-Hsiang Wang, Xingdi Yuan, Tong Wang, Pauline Lucas, Hélène Sauzéon, and Pierre-Yves Oudeyer. 2023 · 2023
Closest in time.
Palm 2 technical report
Original
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. 2023 · 2023
Closest in time.
Pali-x: On scaling up a multilingual vision and language model
Original
Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Carlos Riquelme Ruiz, Sebastian Goodman, Xiao Wang, Yi Tay, et al. 2023 · 2023
Closest in time.
Selection-inference: Exploiting large language models for interpretable logical reasoning
Antonia Creswell, Murray Shanahan, and Irina Higgins. 2023 · 2023
Closest in time.
Khan Academy
Sal Khan. 2023 · 2023
Closest in time.
Decomposed prompting: A modular approach for solving complex tasks
Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. 2023 · 2023
Closest in time.
Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models
Original
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023 · 2023
Closest in time.
Gpt-4 technical report
Original
OpenAI. 2023 · 2023
Closest in time.
Logic-lm: Empowering large language models with symbolic solvers for faithful logical reasoning
Original
Liangming Pan, Alon Albalak, Xinyi Wang, and William Yang Wang. 2023 · 2023
Closest in time.
Testing the general deductive reasoning capacity of large language models using ood examples
Original
Abulhair Saparov, Richard Yuanzhe Pang, Vishakh Padmakumar, Nitish Joshi, Seyed Mehran Kazemi, Najoung Kim, and He He. 2023 · 2023
Closest in time.
Tree of thoughts: Deliberate problem solving with large language models
Original
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. 2023 · 2023
Closest in time.
A multi-modal neural geometric solver with textual clauses parsed from diagram
Original
Ming-Liang Zhang, Fei Yin, and Cheng-Lin Liu. 2023 · 2023
Closest in time.
Least-to-most prompting enables complex reasoning in large language models
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V Le, and Ed H. Chi. 2023 · 2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…