2023

Can ChatGPT Replace Traditional KBQA Models? An In-depth Analysis of the Question Answering Performance of the GPT LLM Family

Tan, Yiming, Min, Dehai, Li, Yu et al.

Understand

ChatGPT is a powerful large language model (LLM) that covers knowledge resources such as Wikipedia and supports natural language question answering using its own knowledge.

Therefore, there is growing interest in exploring whether ChatGPT can replace traditional knowledge-based question answering (KBQA) models.
Although there have been some works analyzing the question answering performance of ChatGPT, there is still a lack of large-scale, comprehensive testing of various types of complex questions to analyze the limitations of the model.
In this paper, we present a framework that follows the black-box testing specifications of CheckList proposed by Ribeiro et.

Built on

Tjong Kim Sang, E.F., De Meulder, F.: Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In: In Proc. NAACL-HLT Conf. pp. 142–147 (2003)
2003
Earlier work this paper cites.
Segura, S., Fraser, G., Sanchez, A.B., Ruiz-Cortés, A.: A survey on metamorphic testing. IEEE Transactions on software engineering 42
2016
Earlier work this paper cites.
Su, Y., Sun, H., Sadler, B., Srivatsa, M., Gür, I., Yan, Z., Yan, X.: On generating characteristic-rich question sets for qa evaluation. In: In Proc. EMNLP Conf. pp. 562–572 (2016)
2016
Earlier work this paper cites.
Yih, W.t., Richardson, M., Meek, C., Chang, M.W., Suh, J.: The value of semantic parse labeling for knowledge base question answering. In: In Proc. ACL Conf. pp. 201–206 (2016)
2016
Earlier work this paper cites.
Ngomo, N.: 9th challenge on question answering over linked data (qald-9). language 7
2018
Earlier work this paper cites.
Talmor, A., Berant, J.: The web as a knowledge-base for answering complex questions. In: In Proc. ACL Conf. pp. 641–651 (2018)
2018
Earlier work this paper cites.
Belinkov, Y., Glass, J.: Analysis methods in neural language processing: A survey. Transactions of the Association for Computational Linguistics 7
2019
Earlier work this paper cites.
Kenton, J.D.M.W.C., Toutanova, L.K.: Bert: Pre-training of deep bidirectional transformers for language understanding. In: In Proc. NAACL-HLT Conf. pp. 4171–4186 (2019)
2019
Earlier work this paper cites.
Petroni, F., Rocktäschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y., Miller, A.: Language models as knowledge bases? In: In Proc. IJCAI Conf. pp. 2463–2473 (2019)
2019
Earlier work this paper cites.
Rychalska, B., Basaj, D., Gosiewska, A., Biecek, P.: Models in the wild: On corruption robustness of neural nlp systems. In: In Proc. ICONIP Conf. pp. 235–247 (2019)
2019
Earlier work this paper cites.
Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., Bowman, S.R.: Superglue: a stickier benchmark for general-purpose language understanding systems. In: In Proc. NeurIPS Conf. pp. 3266–3280 (2019)
2019
Earlier work this paper cites.
Wu, T., Ribeiro, M.T., Heer, J., Weld, D.S.: Errudite: Scalable, reproducible, and testable error analysis. In: In Proc. ACL Conf. pp. 747–763 (2019)
2019
Earlier work this paper cites.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. Advances in neural information processing systems 33
2020
Earlier work this paper cites.
Jiang, Z., Xu, F.F., Araki, J., Neubig, G.: How can we know what language models know? Transactions of the Association for Computational Linguistics 8
2020
Earlier work this paper cites.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P.J.: Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research 21
2020
Earlier work this paper cites.
Ribeiro, M.T., Wu, T., Guestrin, C., Singh, S.: Beyond accuracy: Behavioral testing of nlp models with checklist. In: In Proc. ACL Conf. pp. 4902–4912 (2020)
2020
Earlier work this paper cites.
Gu, Y., Kase, S., Vanni, M., Sadler, B., Liang, P., Yan, X., Su, Y.: Beyond iid: three levels of generalization for question answering on knowledge bases. In: In Proc. WWW Conf. pp. 3477–3488 (2021)
2021
Earlier work this paper cites.
He, H., Choi, J.D.: The stem cell hypothesis: Dilemma behind multi-task learning with transformer encoders. In: In Proc. EMNLP Conf. pp. 5555–5577 (2021)
2021
Earlier work this paper cites.

Similar

Longpre, S., Lu, Y., Daiber, J.: Mkqa: A linguistically diverse benchmark for multilingual open domain question answering. Transactions of the Association for Computational Linguistics 9
2021
Cited alongside, same era.
Pramanik, S., Alabi, J., Saha Roy, R., Weikum, G.: Uniqorn: Unified question answering over rdf knowledge graphs and natural language text. arXiv e-prints pp. arXiv–2108 (2021)
2021
Cited alongside, same era.
Qin, G., Eisner, J.: Learning how to ask: Querying lms with mixtures of soft prompts. In: In Proc. NAACL-HLT Conf. (2021)
2021
Cited alongside, same era.
Rae, J.W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., Young, S., et al.: Scaling language models: Methods, analysis & insights from training gopher. arXiv e-prints pp. arXiv–2112 (2021)
2021
Cited alongside, same era.
Reynolds, L., McDonell, K.: Prompt programming for large language models: Beyond the few-shot paradigm. In: In Proc. CHI EA Conf. pp. 1–7 (2021)
2021
Cited alongside, same era.
Cao, S., Shi, J., Pan, L., Nie, L., Xiang, Y., Hou, L., Li, J., He, B., Zhang, H.: Kqa pro: A dataset with explicit compositional programs for complex question answering over knowledge base. In: In Proc. ACL Conf. pp. 6101–6119 (2022)
2022
Cited alongside, same era.
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H.W., Sutton, C., Gehrmann, S., et al.: Palm: Scaling language modeling with pathways. arXiv e-prints pp. arXiv–2204 (2022)
2022
Cited alongside, same era.
Chung, H.W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, E., Wang, X., Dehghani, M., Brahma, S., et al.: Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416 (2022)
Original
2022
Cited alongside, same era.
Fu, Y., Peng, H., Khot, T.: How does gpt obtain its ability? tracing emergent abilities of language models to their sources. Yao Fu’s Notion (2022)
2022
Cited alongside, same era.
Gu, Y., Su, Y.: Arcaneqa: Dynamic program induction and contextualized encoding for knowledge base question answering. In: In Proc. COLING Conf. pp. 1718–1731 (2022)
2022
Cited alongside, same era.
Hu, X., Wu, X., Shu, Y., Qu, Y.: Logical form generation via multi-task learning for complex question answering over knowledge bases. In: In Proc. ICCL Conf. pp. 1687–1696 (2022)
2022
Cited alongside, same era.
Kojima, T., Gu, S.S., Reid, M., Matsuo, Y., Iwasawa, Y.: Large language models are zero-shot reasoners. arXiv preprint arXiv:2205.11916 (2022)
Original
2022
Cited alongside, same era.

Then

Wei, J., Wang, X., Schuurmans, D., Bosma, M., Chi, E., Le, Q., Zhou, D.: Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903 (2022)
Original
2022
Later among the works it cites.
Ye, X., Yavuz, S., Hashimoto, K., Zhou, Y., Xiong, C.: Rng-kbqa: Generation augmented iterative ranking for knowledge base question answering. In: In Proc. ACL Conf. pp. 6032–6043 (2022)
2022
Later among the works it cites.
Bai, Y., Ying, J., Cao, Y., Lv, X., He, Y., Wang, X., Yu, J., Zeng, K., Xiao, Y., Lyu, H., et al.: Benchmarking foundation models with language-model-as-an-examiner. arXiv preprint arXiv:2306.04181 (2023)
Original
2023
Closest in time.
Bang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., Lovenia, H., Ji, Z., Yu, T., Chung, W., et al.: A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv e-prints pp. arXiv–2302 (2023)
2023
Closest in time.
Chang, Y., Wang, X., Wang, J., Wu, Y., Zhu, K., Chen, H., Yang, L., Yi, X., Wang, C., Wang, Y., et al.: A survey on evaluation of large language models. arXiv preprint arXiv:2307.03109 (2023)
Original
2023
Closest in time.
Chen, X., Ye, J., Zu, C., Xu, N., Zheng, R., Peng, M., Zhou, J., Gui, T., Zhang, Q., Huang, X.: How robust is gpt-3.5 to predecessors? a comprehensive study on language understanding tasks. arXiv e-prints pp. arXiv–2303 (2023)
2023
Closest in time.
Fu, C., Chen, P., Shen, Y., Qin, Y., Zhang, M., Lin, X., Qiu, Z., Lin, W., Yang, J., Zheng, X., et al.: Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394 (2023)
Original
2023
Closest in time.
Huang, F., Kwak, H., An, J.: Is chatgpt better than human annotators? potential and limitations of chatgpt in explaining implicit hate speech. arXiv e-prints pp. arXiv–2302 (2023)
2023
Closest in time.
Kocoń, J., Cichecki, I., Kaszyca, O., Kochanek, M., Szydło, D., Baran, J., Bielaniewicz, J., Gruza, M., Janz, A., Kanclerz, K., et al.: Chatgpt: Jack of all trades, master of none. arXiv e-prints pp. arXiv–2302 (2023)
2023
Closest in time.
Lyu, C., Xu, J., Wang, L.: New trends in machine translation using large language models: Case examples with chatgpt. arXiv preprint arXiv:2305.01181 (2023)
Original
2023
Closest in time.
Omar, R., Mangukiya, O., Kalnis, P., Mansour, E.: Chatgpt versus traditional question answering for knowledge graphs: Current status and future directions towards knowledge graph chatbots. arXiv e-prints pp. arXiv–2302 (2023)
2023
Closest in time.
OpenAI: Gpt-4 technical report (2023)
2023
Closest in time.
Qin, C., Zhang, A., Zhang, Z., Chen, J., Yasunaga, M., Yang, D.: Is chatgpt a general-purpose natural language processing task solver? arXiv e-prints pp. arXiv–2302 (2023)
2023
Closest in time.
Wang, J., Liang, Y., Meng, F., Li, Z., Qu, J., Zhou, J.: Cross-lingual summarization via chatgpt. arXiv e-prints pp. arXiv–2302 (2023)
2023
Closest in time.
Wang, S., Scells, H., Koopman, B., Zuccon, G.: Can chatgpt write a good boolean query for systematic review literature search? arXiv e-prints pp. arXiv–2302 (2023)
2023
Closest in time.
Zhong, Q., Ding, L., Liu, J., Du, B., Tao, D.: Can chatgpt understand too? a comparative study on chatgpt and fine-tuned bert. arXiv e-prints pp. arXiv–2302 (2023)
2023
Closest in time.
Zhu, K., Wang, J., Zhou, J., Wang, Z., Chen, H., Wang, Y., Yang, L., Ye, W., Gong, N.Z., Zhang, Y., et al.: Promptbench: Towards evaluating the robustness of large language models on adversarial prompts. arXiv preprint arXiv:2306.04528 (2023)
Original
2023
Closest in time.
Zhuo, T.Y., Huang, Y., Chen, C., Xing, Z.: Exploring ai ethics of chatgpt: A diagnostic analysis. arXiv e-prints pp. arXiv–2301 (2023)
2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…