Built on

R. Jia and P. Liang, Adversarial examples for evaluating reading comprehension systems, arXiv preprint arXiv:1707.07328 (2017)
Original
2017
Earlier work this paper cites.
D. Gunning and D. Aha, Darpa’s explainable artificial intelligence (xai) program, AI magazine 40
2019
Earlier work this paper cites.
S. Mei, T. Misiakiewicz, and A. Montanari, Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit, in Conference on learning theory (PMLR, 2019) pp. 2388–2464
2019
Earlier work this paper cites.
R. McCoy, Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference, arXiv preprint arXiv:1902.01007 (2019)
Original
2019
Earlier work this paper cites.
Y. Blumenfeld, D. Gilboa, and D. Soudry, A mean field theory of quantized deep networks: The quantization-depth trade-off, Advances in Neural Information Processing Systems 32
2019
Earlier work this paper cites.
K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. , Training verifiers to solve math word problems, arXiv preprint arXiv:2110.14168 (2021)
Original
2021
Earlier work this paper cites.
Y. Feng and Y. Tu, Phases of learning dynamics in artificial neural networks: with or without mislabeled data, arXiv preprint arXiv:2101.06509 (2021)
Original
2021
Earlier work this paper cites.
B. Wang, X. Jiang, G. Huo, C. Su, D. Yan, and Z. Zheng, Key-point interpolation: A sparse data interpolation algorithm based on b-splines, in Journal of Physics: Conference Series , Vol. 2068 (IOP Publishing, 2021) p. 012010
2021
Earlier work this paper cites.
J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al. , Chain-of-thought prompting elicits reasoning in large language models, Advances in neural information processing systems 35
2022
Earlier work this paper cites.
M. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, et al. , Challenging big-bench tasks and whether chain-of-thought can solve them, arXiv preprint arXiv:2210.09261 (2022)
Original
2022
Earlier work this paper cites.
X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, and D. Zhou, Self-consistency improves chain of thought reasoning in language models, arXiv preprint arXiv:2203.11171 (2022)
Original
2022
Earlier work this paper cites.
Y. Razeghi, R. L. Logan IV, M. Gardner, and S. Singh, Impact of pretraining term frequencies on few-shot reasoning, arXiv preprint arXiv:2202.07206 (2022)
Original
2022
Earlier work this paper cites.
K. Valmeekam, A. Olmo, S. Sreedharan, and S. Kambhampati, Large language models still can’t plan (a benchmark for llms on planning and reasoning about change), in NeurIPS 2022 Foundation Models for Decision Making Workshop (2022)
2022
Earlier work this paper cites.

Similar

I. Dasgupta, A. K. Lampinen, S. C. Chan, A. Creswell, D. Kumaran, J. L. McClelland, and F. Hill, Language models show human-like content effects on reasoning, arXiv preprint arXiv:2207.07051 2
Original
2022
Cited alongside, same era.
R. OpenAI, Gpt-4 technical report. arxiv 2303.08774, View in Article 2
2023
Cited alongside, same era.
H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. , Llama: Open and efficient foundation language models, arXiv preprint arXiv:2302.13971 (2023)
Original
2023
Cited alongside, same era.
G. Team, R. Anil, S. Borgeaud, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican, et al. , Gemini: a family of highly capable multimodal models, arXiv preprint arXiv:2312.11805 (2023)
Original
2023
Cited alongside, same era.
S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg, et al. , Sparks of artificial general intelligence: Early experiments with gpt-4, arXiv preprint arXiv:2303.12712 (2023)

Then

Y. Wan, W. Wang, Y. Yang, Y. Yuan, J.-t. Huang, P. He, W. Jiao, and M. R. Lyu, A & b== b & a: Triggering logical reasoning failures in large language models, arXiv preprint arXiv:2401.00757 (2024)
Original
2024
Later among the works it cites.
Z. Z. Chen, J. Ma, X. Zhang, N. Hao, A. Yan, A. Nourbakhsh, X. Yang, J. McAuley, L. Petzold, and W. Y. Wang, A survey on large language models for critical societal domains: Finance, healthcare, and law, arXiv preprint arXiv:2405.01769 (2024)
Original
2024
Later among the works it cites.
S. K. Radha, Y. N. Jelyani, A. Ghukasyan, and O. Goktas, Iteration of thought: Leveraging inner dialogue for autonomous large language model reasoning, arXiv preprint arXiv:2409.12618 (2024)
Original
2024
Later among the works it cites.
N. Dziri, X. Lu, M. Sclar, X. L. Li, L. Jiang, B. Y. Lin, S. Welleck, P. West, C. Bhagavatula, R. Le Bras, et al. , Faith and fate: Limits of transformers on compositionality, Advances in Neural Information Processing Systems 36
2024
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2025

On the Reasoning Capacity of AI Models and How to Quantify It

Radha, Santosh Kumar, Goktas, Oktay

Understand

Recent advances in Large Language Models (LLMs) have intensified the debate surrounding the fundamental nature of their reasoning capabilities.

While achieving high performance on benchmarks such as GPQA and MMLU, these models exhibit limitations in more complex reasoning tasks, highlighting the need for more rigorous evaluation methodologies.
We propose a novel phenomenological approach that goes beyond traditional accuracy metrics to probe the underlying mechanisms of model behavior, establishing a framework that could broadly impact how we analyze and understand AI systems.
Using positional bias in multiple-choice reasoning tasks as a case study, we demonstrate how systematic perturbations can reveal fundamental aspects of model decision-making.

Built on

R. Jia and P. Liang, Adversarial examples for evaluating reading comprehension systems, arXiv preprint arXiv:1707.07328 (2017)
Original
2017
Earlier work this paper cites.
D. Gunning and D. Aha, Darpa’s explainable artificial intelligence (xai) program, AI magazine 40
2019
Earlier work this paper cites.
S. Mei, T. Misiakiewicz, and A. Montanari, Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit, in Conference on learning theory (PMLR, 2019) pp. 2388–2464
2019
Earlier work this paper cites.
R. McCoy, Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference, arXiv preprint arXiv:1902.01007 (2019)
Original
2019
Earlier work this paper cites.
Y. Blumenfeld, D. Gilboa, and D. Soudry, A mean field theory of quantized deep networks: The quantization-depth trade-off, Advances in Neural Information Processing Systems 32
2019
Earlier work this paper cites.
K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. , Training verifiers to solve math word problems, arXiv preprint arXiv:2110.14168 (2021)
Original
2021
Earlier work this paper cites.
Y. Feng and Y. Tu, Phases of learning dynamics in artificial neural networks: with or without mislabeled data, arXiv preprint arXiv:2101.06509 (2021)
Original
2021
Earlier work this paper cites.
B. Wang, X. Jiang, G. Huo, C. Su, D. Yan, and Z. Zheng, Key-point interpolation: A sparse data interpolation algorithm based on b-splines, in Journal of Physics: Conference Series , Vol. 2068 (IOP Publishing, 2021) p. 012010
2021
Earlier work this paper cites.
J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al. , Chain-of-thought prompting elicits reasoning in large language models, Advances in neural information processing systems 35
2022
Earlier work this paper cites.
M. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, et al. , Challenging big-bench tasks and whether chain-of-thought can solve them, arXiv preprint arXiv:2210.09261 (2022)
Original
2022
Earlier work this paper cites.
X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, and D. Zhou, Self-consistency improves chain of thought reasoning in language models, arXiv preprint arXiv:2203.11171 (2022)
Original
2022
Earlier work this paper cites.
Y. Razeghi, R. L. Logan IV, M. Gardner, and S. Singh, Impact of pretraining term frequencies on few-shot reasoning, arXiv preprint arXiv:2202.07206 (2022)
Original
2022
Earlier work this paper cites.
K. Valmeekam, A. Olmo, S. Sreedharan, and S. Kambhampati, Large language models still can’t plan (a benchmark for llms on planning and reasoning about change), in NeurIPS 2022 Foundation Models for Decision Making Workshop (2022)
2022
Earlier work this paper cites.

Similar

I. Dasgupta, A. K. Lampinen, S. C. Chan, A. Creswell, D. Kumaran, J. L. McClelland, and F. Hill, Language models show human-like content effects on reasoning, arXiv preprint arXiv:2207.07051 2
Original
2022
Cited alongside, same era.
R. OpenAI, Gpt-4 technical report. arxiv 2303.08774, View in Article 2
2023
Cited alongside, same era.
H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. , Llama: Open and efficient foundation language models, arXiv preprint arXiv:2302.13971 (2023)
Original
2023
Cited alongside, same era.
G. Team, R. Anil, S. Borgeaud, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican, et al. , Gemini: a family of highly capable multimodal models, arXiv preprint arXiv:2312.11805 (2023)
Original
2023
Cited alongside, same era.
S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg, et al. , Sparks of artificial general intelligence: Early experiments with gpt-4, arXiv preprint arXiv:2303.12712 (2023)

Then

Y. Wan, W. Wang, Y. Yang, Y. Yuan, J.-t. Huang, P. He, W. Jiao, and M. R. Lyu, A & b== b & a: Triggering logical reasoning failures in large language models, arXiv preprint arXiv:2401.00757 (2024)
Original
2024
Later among the works it cites.
Z. Z. Chen, J. Ma, X. Zhang, N. Hao, A. Yan, A. Nourbakhsh, X. Yang, J. McAuley, L. Petzold, and W. Y. Wang, A survey on large language models for critical societal domains: Finance, healthcare, and law, arXiv preprint arXiv:2405.01769 (2024)
Original
2024
Later among the works it cites.
S. K. Radha, Y. N. Jelyani, A. Ghukasyan, and O. Goktas, Iteration of thought: Leveraging inner dialogue for autonomous large language model reasoning, arXiv preprint arXiv:2409.12618 (2024)
Original
2024
Later among the works it cites.
N. Dziri, X. Lu, M. Sclar, X. L. Li, L. Jiang, B. Y. Lin, S. Welleck, P. West, C. Bhagavatula, R. Le Bras, et al. , Faith and fate: Limits of transformers on compositionality, Advances in Neural Information Processing Systems 36
2024
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Original

2023

Cited alongside, same era.

D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman, Gpqa: A graduate-level google-proof q&a benchmark, arXiv preprint arXiv:2311.12022 (2023)

Original

2023

Cited alongside, same era.

Z. Wu, L. Qiu, A. Ross, E. Akyürek, B. Chen, B. Wang, N. Kim, J. Andreas, and Y. Kim, Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks, arXiv preprint arXiv:2307.02477 (2023)

Original

2023

Cited alongside, same era.

G. Gendron, Q. Bao, M. Witbrock, and G. Dobbie, Large language models are not strong abstract reasoners, arXiv preprint arXiv:2305.19555 (2023)

Original

2023

Cited alongside, same era.

S. Tovey, S. Krippendorf, K. Nikolaou, and C. Holm, Towards a phenomenological understanding of neural networks: data, Machine Learning: Science and Technology 4

2023

Cited alongside, same era.

P. Pezeshkpour and E. Hruschka, Large language models sensitivity to the order of options in multiple-choice questions, arXiv preprint arXiv:2308.11483 (2023)

Original

2023

Cited alongside, same era.

C. Zheng, H. Zhou, F. Meng, J. Zhou, and M. Huang, Large language models are not robust multiple choice selectors, in The Twelfth International Conference on Learning Representations (2023)

2023

Cited alongside, same era.

D. Huang, Q. Bu, J. Zhang, X. Xie, J. Chen, and H. Cui, Bias assessment and mitigation in llm-based code generation, arXiv preprint arXiv:2309.14345 (2023)

Original

2023

Cited alongside, same era.

B. Jiang, Y. Xie, Z. Hao, X. Wang, T. Mallick, W. J. Su, C. J. Taylor, and D. Roth, A peek into token bias: Large language models are not yet genuine reasoners, arXiv preprint arXiv:2406.11050 (2024)

Original

2024