2023

Context Matters: A Strategy to Pre-train Language Model for Science Education

Liu, Zhengliang, He, Xinyu, Liu, Lei et al.

Understand

This study aims at improving the performance of scoring student responses in science education automatically.

BERT-based language models have shown significant superiority over traditional NLP models in various language-related tasks.
However, science writing of students, including argumentation and explanation, is domain-specific.
In addition, the language used by students is different from the language in journals and Wikipedia, which are training sources of BERT and its existing variants.

Novak, A.M., McNeill, K.L., Krajcik, J.S.: Helping students write scientific explanations. Science Scope 33
2009
Earlier work this paper cites.
Ha, M., Nehm, R.H.: The impact of misspelled words on automated computer scoring: A case study of scientific explanations. Journal of Science Education and Technology 25
2016
Earlier work this paper cites.
Litman, D.: Natural language processing for enhancing teaching and learning. In: Thirtieth AAAI conference on artificial intelligence (2016)
2016
Earlier work this paper cites.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information processing systems 30
2017
Earlier work this paper cites.
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
Original
2018
Earlier work this paper cites.
Araci, D.: Finbert: Financial sentiment analysis with pre-trained language models. arXiv preprint arXiv:1908.10063 (2019)
Original
2019
Earlier work this paper cites.

Beltagy, I., Lo, K., Cohan, A.: Scibert: A pretrained language model for scientific text. arXiv preprint arXiv:1903.10676 (2019)
Original
2019
Cited alongside, same era.
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C.H., Kang, J.: Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics 36
2020
Cited alongside, same era.
Zhai, X., C Haudek, K., Shi, L., H Nehm, R., Urban-Lurain, M.: From substitution to redefinition: A framework of machine learning-based science assessment. Journal of Research in Science Teaching 57
2020
Cited alongside, same era.
Zhai, X., Yin, Y., Pellegrino, J.W., Haudek, K.C., Shi, L.: Applying machine learning in science assessment: a systematic review. Studies in Science Education 56
2020
Cited alongside, same era.
Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., Poon, H.: Domain-specific language model pretraining for biomedical natural language processing. ACM Transactions on Computing for Healthcare (HEALTH) 3
2021
Cited alongside, same era.
Haudek, K.C., Zhai, X.: Exploring the effect of assessment construct complexity on machine learning scoring of argumentation (2021)
2021
Cited alongside, same era.

Rezayi, S., Dai, H., Liu, Z., Wu, Z., Hebbar, A., Burns, A.H., Zhao, L., Zhu, D., Li, Q., Liu, W., et al.: Clinicalradiobert: Knowledge-infused few shot learning for clinical notes named entity recognition. In: International Workshop on Machine Learning in Medical Imaging. pp. 269–278. Springer (2022)
2022
Later among the works it cites.
Rezayi, S., Liu, Z., Wu, Z., Dhakal, C., Ge, B., Zhen, C., Liu, T., Li, S.: Agribert: knowledge-infused agricultural language models for matching food and nutrition. In: IJCAI. IJCAI (2022)
2022
Later among the works it cites.
Zhai, X., Haudek, K.C., Ma, W.: Assessing argumentation using machine learning and cognitive diagnostic modeling. Research in Science Education pp. 1–20 (2022)
2022
Later among the works it cites.
Zhai, X., He, P., Krajcik, J.: Applying machine learning to automatically assess scientific models. Journal of Research in Science Teaching 59
2022
Later among the works it cites.
Amerman, H., Zhai, X., Latif, E., He, P., Krajcik, J.: Does transformer deep learning yield more accurate sores on student written explanations than traditional machine learning? In: Paper submitted to the Annual Meeting of the American Educational Research Association. Chicago (2023)
2023
Closest in time.

alphaXiv searches the wider corpus for related work and actual follow-ups.

alphaXiv is searching for related work…

Reading the bibliography…