Fetching the paper…
Reading the bibliography…
Large language models (LLMs) achieve good performance on challenging reasoning benchmarks, yet could also make basic reasoning mistakes.
What is the Name of this Book?
Smullyan, R · 1978
Earlier work this paper cites.
Meta-logical problems: Knights, knaves, and rips
Johnson-Laird, P. N. and Byrne, R. M · 1990
Earlier work this paper cites.
Fine-grained analysis of sentence embeddings using auxiliary prediction tasks
Adi, Y., Kermany, E., Belinkov, Y., Lavi, O., and Goldberg, Y · 2017
Earlier work this paper cites.
Overfitting or perfect fitting? Risk bounds for classification and regression rules that interpolate
Belkin, M., Hsu, D. J., and Mitra, P · 2018
Earlier work this paper cites.
What you can cram into a single vector: Probing sentence embeddings for linguistic properties
Conneau, A., Kruszewski, G., Lample, G., Barrault, L., and Baroni, M · 2018
Earlier work this paper cites.
Designing and interpreting probes with control tasks
Hewitt, J. and Liang, P · 2019
Earlier work this paper cites.
Transformers as soft reasoners over language
Clark, P., Tafjord, O., and Richardson, K · 2020
Earlier work this paper cites.
The pile: An 800gb dataset of diverse text for language modeling
Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al · 2020
Earlier work this paper cites.
Is bert really robust? natural language attack on text classification and entailment
Jin, D., Jin, Z., Zhou, J. T., and Szolovits, P · 2020
Earlier work this paper cites.
Harmless interpolation of noisy data in regression
Muthukumar, V., Vodrahalli, K., Subramanian, V., and Sahai, A · 2020
Earlier work this paper cites.
Adversarial NLI: A new benchmark for natural language understanding
Nie, Y., Williams, A., Dinan, E., Bansal, M., Weston, J., and Kiela, D · 2020
Earlier work this paper cites.
Deep learning: a statistical viewpoint
Bartlett, P. L., Montanari, A., and Rakhlin, A · 2021
Earlier work this paper cites.
Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolation
Belkin, M · 2021
Earlier work this paper cites.
Extracting training data from large language models
Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., et al · 2021
Earlier work this paper cites.
Adversarial GLUE: A multi-task benchmark for robustness evaluation of language models
Wang, B., Xu, C., Wang, S., Gan, Z., Cheng, Y., Gao, J., Awadallah, A. H., and Li, B · 2021
Earlier work this paper cites.
Calibrate before use: Improving few-shot performance of language models
Zhao, T., Wallace, E., Feng, S., Klein, D., and Singh, S · 2021
Earlier work this paper cites.
Deduplicating training data makes language models better
Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Burch, C., and Carlini, N · 2022
Earlier work this paper cites.
Data contamination: From memorization to exploitation
Magar, I. and Schwartz, R · 2022
Earlier work this paper cites.
Grokking: Generalization beyond overfitting on small algorithmic datasets
Power, A., Burda, Y., Edwards, H., Babuschkin, I., and Misra, V · 2022
Earlier work this paper cites.
Impact of pretraining term frequencies on few-shot numerical reasoning
Razeghi, Y., Logan IV, R. L., Gardner, M., and Singh, S · 2022
Earlier work this paper cites.
Memorization without overfitting: Analyzing the training dynamics of large language models
Tirumala, K., Markosyan, A., Zettlemoyer, L., and Aghajanyan, A · 2022
Earlier work this paper cites.
Chain-of-thought prompting elicits reasoning in large language models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al · 2022
Earlier work this paper cites.
Star: Bootstrapping reasoning with reasoning
Zelikman, E., Wu, Y., Mu, J., and Goodman, N · 2022
Earlier work this paper cites.
Quantifying memorization across neural language models
Carlini, N., Ippolito, D., Jagielski, M., Lee, K., Tramer, F., and Zhang, C · 2023
Earlier work this paper cites.
Implicit chain of thought reasoning via knowledge distillation
Deng, Y., Prasad, K., Fernandez, R., Smolensky, P., Chaudhary, V., and Shieber, S · 2023
Earlier work this paper cites.
Data contamination quiz: A tool to detect and estimate contamination in large language models
Golchin, S. and Surdeanu, M · 2023
Earlier work this paper cites.
SoK: memorization in general-purpose large language models
Hartmann, V., Suri, A., Bindschaedler, V., Evans, D., Tople, S., and West, R · 2023
Cited alongside, same era.
Large language models are reasoning teachers
Ho, N., Schmid, L., and Yun, S.-Y · 2023
Cited alongside, same era.
Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes
Hsieh, C.-Y., Li, C.-L., Yeh, C.-K., Nakhost, H., Fujii, Y., Ratner, A., Krishna, R., Lee, C.-Y., and Pfister, T · 2023
Cited alongside, same era.
Copyright violations and large language models
Karamolegkou, A., Li, J., Zhou, L., and Søgaard, A · 2023
Cited alongside, same era.
The cot collection: Improving zero-shot and few-shot learning of language models via chain-of-thought fine-tuning
Kim, S., Joo, S. J., Kim, D., Jang, J., Ye, S., Shin, J., and Seo, M · 2023
Cited alongside, same era.
BoardgameQA: A dataset for natural language reasoning with contradictory information
Kazemi, M., Yuan, Q., Bhatia, D., Kim, N., Xu, X., Imbrasaite, V., and Ramachandran, D · 2024
Closest in time.
A mechanistic understanding of alignment algorithms: A case study on DPO and toxicity
Lee, A., Bai, X., Pres, I., Wattenberg, M., Kummerfeld, J. K., and Mihalcea, R · 2024
Closest in time.
Math Olympiad becomes easier for AI; Common sense is still hard., 2024
Lin, B. Y · 2024
Closest in time.
ZebraLogic: benchmarking the logical reasoning ability of language models, 2024
Lin, B. Y., Bras, R. L., and Choi, Y · 2024
Closest in time.
Embers of autoregression show how large language models are shaped by the problem they are trained to solve
McCoy, R. T., Yao, S., Friedman, D., Hardy, M. D., and Griffiths, T. L · 2024
Closest in time.
Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Analyzing leakage of personally identifiable information in language models
Lukas, N., Salem, A., Sim, R., Tople, S., Wutschitz, L., and Zanella-Béguelin, S · 2023
Cited alongside, same era.
Grokking of hierarchical structure in vanilla transformers
Murty, S., Sharma, P., Andreas, J., and Manning, C. D · 2023
Cited alongside, same era.
Logic-lm: Empowering large language models with symbolic solvers for faithful logical reasoning
Pan, L., Albalak, A., Wang, X., and Wang, W. Y · 2023
Cited alongside, same era.
To the cutoff… and beyond? A longitudinal perspective on LLM data contamination
Roberts, M., Thakur, H., Herlihy, C., White, C., and Dooley, S · 2023
Cited alongside, same era.
Leveraging large language models for multiple choice question answering
Robinson, J. and Wingate, D · 2023
Cited alongside, same era.
Language models are greedy reasoners: A systematic formal analysis of chain-of-thought
Saparov, A. and He, H · 2023
Cited alongside, same era.
Large language models are better reasoners with self-verification
Weng, Y., Zhu, M., Xia, F., Li, B., He, S., Liu, S., Sun, B., Liu, K., and Zhao, J · 2023
Cited alongside, same era.
Mirzadeh, I., Alizadeh, K., Shahrokhi, H., Tuzel, O., Bengio, S., and Farajtabar, M · 2024
Closest in time.
Liar, liar, logical mire: A benchmark for suppositional reasoning in large language models
Mondorf, P. and Plank, B · 2024
Closest in time.
Nezhurina, M., Cipolina-Kun, L., Cherti, M., and Jitsev, J · 2024
Closest in time.
Proving test set contamination in black box language models
Oren, Y., Meister, N., Chatterji, N., Ladhak, F., and Hashimoto, T. B · 2024
Closest in time.
LogicBench: towards systematic evaluation of logical reasoning ability of large language models
Parmar, M., Patel, N., Varshney, N., Nakamura, M., Luo, M., Mashetty, S., Mitra, A., and Baral, C · 2024
Closest in time.
Prabhakar, A., Griffiths, T. L., and McCoy, R. T · 2024
Closest in time.
Recite, reconstruct, recollect: Memorization in lms as a multifaceted phenomenon
Prashanth, U. S., Deng, A., O’Brien, K., SV, J., Khan, M. A., Borkar, J., Choquette-Choo, C. A., Fuehne, J. R., Biderman, S., Ke, T., et al · 2024
Closest in time.
Puerto, H., Chubakov, T., Zhu, X., Madabushi, H. T., and Gurevych, I · 2024
Closest in time.
Sat solver — Wikipedia, the free encyclopedia, 2024
SAT solver · 2024
Closest in time.
Detecting pretraining data from large language models
Shi, W., Ajith, A., Xia, M., Huang, Y., Liu, D., Blevins, T., Chen, D., and Zettlemoyer, L · 2024
Closest in time.
Functional benchmarks for robust evaluation of reasoning performance, and the reasoning gap
Srivastava, S., PV, A., Menon, S., Sukumar, A., Philipose, A., Prince, S., Thomas, S., et al · 2024
Closest in time.
The instruction hierarchy: Training LLMs to prioritize privileged instructions
Wallace, E., Xiao, K., Leike, R., Weng, L., Heidecke, J., and Beutel, A · 2024
Closest in time.
Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks
Wu, Z., Qiu, L., Ross, A., Akyürek, E., Chen, B., Wang, B., Kim, N., Andreas, J., and Kim, Y · 2024
Closest in time.
Benchmarking benchmark leakage in large language models
Xu, R., Wang, Z., Fan, R.-Z., and Liu, P · 2024
Closest in time.
Data contamination can cross language barriers
Yao, F., Zhuang, Y., Sun, Z., Xu, S., Kumar, A., and Shang, J · 2024
Closest in time.
Physics of language models: Part 2.1, grade-school math and the hidden reasoning process
Ye, T., Xu, Z., Li, Y., and Allen-Zhu, Z · 2024
Closest in time.
A careful examination of large language model performance on grade school arithmetic
Zhang, H., Da, J., Lee, D., Robinson, V., Wu, C., Song, W., Zhao, T., Raja, P., Slack, D., Lyu, Q., et al · 2024
Closest in time.
Revisiting the self-consistency challenges in multi-choice question formats for large language model evaluation
Zhou, W., Wang, Q., Xu, M., Chen, M., and Duan, X · 2024
Closest in time.
Dyval: Graph-informed dynamic evaluation of large language models
Zhu, K., Chen, J., Wang, J., Gong, N. Z., Yang, D., and Xie, X · 2024
Closest in time.
Fool your (vision and) language model with embarrassingly simple permutations
Zong, Y., Yu, T., Zhao, B., Chavhan, R., and Hospedales, T · 2024
Closest in time.