2023

AcademicGPT: Empowering Academic Research

Wei, Shufa, Xu, Xiaolong, Qi, Xianbiao et al.

Understand

Large Language Models (LLMs) have demonstrated exceptional capabilities across various natural language processing tasks.

Yet, many of these advanced LLMs are tailored for broad, general-purpose applications.
In this technical report, we introduce AcademicGPT, designed specifically to empower academic research.
AcademicGPT is a continual training model derived from LLaMA2-70B.

Built on

Language models are few-shot learners
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · 1901
Earlier work this paper cites.
A study of bfloat16 for deep learning training
Original
Kalamkar, D., Mudigere, D., Mellempudi, N., Das, D., Banerjee, K., Avancha, S., Vooturi, D. T., Jammalamadaka, N., Huang, J., Yuen, H., et al. (2019) · 1905
Earlier work this paper cites.
Ernie: Enhanced language representation with informative entities
Original
Zhang, Z., Han, X., Liu, Z., Jiang, X., Sun, M., and Liu, Q. (2019) · 1905
Earlier work this paper cites.
Pubmedqa: A dataset for biomedical research question answering
Original
Jin, Q., Dhingra, B., Liu, Z., Cohen, W. W., and Lu, X. (2019) · 1909
Earlier work this paper cites.
Gradient-based learning applied to document recognition
LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. (1998) · 1998
Earlier work this paper cites.
Glu variants improve transformer
Original
Shazeer, N. (2020) · 2002
Earlier work this paper cites.
Measuring massive multitask language understanding
Original
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. (2020) · 2009
Earlier work this paper cites.
Neural machine translation of rare words with subword units
Original
Sennrich, R., Haddow, B., and Birch, A. (2015) · 2015
Earlier work this paper cites.
Layer normalization
Original
Ba, J. L., Kiros, J. R., and Hinton, G. E. (2016) · 2016
Earlier work this paper cites.
Sgdr: Stochastic gradient descent with warm restarts
Original
Loshchilov, I. and Hutter, F. (2016) · 2016
Earlier work this paper cites.
Decoupled weight decay regularization
Original
Loshchilov, I. and Hutter, F. (2017) · 2017
Earlier work this paper cites.
Semi-supervised sequence tagging with bidirectional language models
Original
Peters, M. E., Ammar, W., Bhagavatula, C., and Power, R. (2017) · 2017
Earlier work this paper cites.
Attention is all you need
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. (2017) · 2017
Earlier work this paper cites.
Bert: Pre-training of deep bidirectional transformers for language understanding
Original
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018) · 2018
Earlier work this paper cites.
Improving language understanding by generative pre-training
Radford, A., Narasimhan, K., Salimans, T., Sutskever, I., et al. (2018) · 2018
Earlier work this paper cites.
Language models are unsupervised multitask learners
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. (2019) · 2019
Earlier work this paper cites.
Root mean square layer normalization
Zhang, B. and Sennrich, R. (2019) · 2019
Earlier work this paper cites.
The pile: An 800gb dataset of diverse text for language modeling
Original
Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. (2020) · 2020
Earlier work this paper cites.
Zero: Memory optimizations toward training trillion parameter models
Rajbhandari, S., Rasley, J., Ruwase, O., and He, Y. (2020) · 2020
Earlier work this paper cites.
Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters
Rasley, J., Rajbhandari, S., Ruwase, O., and He, Y. (2020) · 2020
Earlier work this paper cites.
Glm: General language model pretraining with autoregressive blank infilling
Original
Du, Z., Qian, Y., Liu, X., Ding, M., Qiu, J., Yang, Z., and Tang, J. (2021) · 2021
Earlier work this paper cites.
Lifelong pretraining: Continually adapting language models to emerging corpora
Original
Jin, X., Zhang, D., Zhu, H., Xiao, W., Li, S.-W., Wei, X., Arnold, A., and Ren, X. (2021) · 2021
Earlier work this paper cites.
Train short, test long: Attention with linear biases enables input length extrapolation
Original
Press, O., Smith, N. A., and Lewis, M. (2021) · 2021
Earlier work this paper cites.
Scaling language models: Methods, analysis & insights from training gopher
Original
Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., Young, S., et al. (2021) · 2021
Earlier work this paper cites.
Roformer: Enhanced transformer with rotary position embedding
Original
Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B., and Liu, Y. (2021) · 2021
Earlier work this paper cites.

Similar

Ernie 3.0: Large-scale knowledge enhanced pre-training for language understanding and generation
Original
Sun, Y., Wang, S., Feng, S., Ding, S., Pang, C., Shang, J., Liu, J., Chen, X., Zhao, Y., Lu, Y., et al. (2021) · 2021
Cited alongside, same era.
Wudaocorpora: A super large-scale chinese corpora for pre-training language models
Yuan, S., Zhao, H., Du, Z., Ding, M., Liu, X., Cen, Y., Zou, X., Yang, Z., and Tang, J. (2021) · 2021
Cited alongside, same era.
Training a helpful and harmless assistant with reinforcement learning from human feedback
Original
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. (2022) · 2022
Cited alongside, same era.
Gpt-neox-20b: An open-source autoregressive language model
Original
Black, S., Biderman, S., Hallahan, E., Anthony, Q., Gao, L., Golding, L., He, H., Leahy, C., McDonell, K., Phang, J., et al. (2022) · 2022
Cited alongside, same era.
Palm: Scaling language modeling with pathways
Original
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · 2022
Cited alongside, same era.
Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity
Fedus, W., Zoph, B., and Shazeer, N. (2022) · 2022
Cited alongside, same era.
Training compute-optimal large language models
Original
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022) · 2022
Cited alongside, same era.
Mrkl systems: A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoning
Original
Karpas, E., Abend, O., Belinkov, Y., Lenz, B., Lieber, O., Ratner, N., Shoham, Y., Bata, H., Levine, Y., Leyton-Brown, K., et al. (2022) · 2022
Cited alongside, same era.
Continual pre-training of language models
Ke, Z., Shao, Y., Lin, H., Konishi, T., Kim, G., and Liu, B. (2022) · 2022
Cited alongside, same era.
The bigscience roots corpus: A 1.6 tb composite multilingual dataset
Laurençon, H., Saulnier, L., Wang, T., Akiki, C., Villanova del Moral, A., Le Scao, T., Von Werra, L., Mou, C., González Ponferrada, E., Nguyen, H., et al. (2022) · 2022
Cited alongside, same era.
Biogpt: generative pre-trained transformer for biomedical text generation and mining
Luo, R., Sun, L., Xia, Y., Qin, T., Zhang, S., Poon, H., and Liu, T.-Y. (2022) · 2022
Cited alongside, same era.
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · 2022
Cited alongside, same era.

Then

Chemcrow: Augmenting large-language models with chemistry tools
Original
Bran, A. M., Cox, S., White, A. D., and Schwaller, P. (2023) · 2023
Closest in time.
Sparks of artificial general intelligence: Early experiments with gpt-4
Original
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023) · 2023
Closest in time.
Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality
Chiang, W.-L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J. E., Stoica, I., and Xing, E. P. (2023) · 2023
Closest in time.
Chatlaw: Open-source legal large language model with integrated external knowledge bases
Original
Cui, J., Li, Z., Yan, Y., Chen, B., and Yuan, L. (2023) · 2023
Closest in time.
Flashattention-2: Faster attention with better parallelism and work partitioning
Original
Dao, T. (2023) · 2023
Closest in time.
Continual pre-training of large language models: How to (re) warm your model?
Original
Gupta, K., Thérien, B., Ibrahim, A., Richter, M. L., Anthony, Q., Belilovsky, E., Rish, I., and Lesort, T. (2023) · 2023
Closest in time.
C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models
Original
Huang, Y., Bai, Y., Zhu, Z., Zhang, J., Zhang, J., Su, T., Liu, J., Lv, C., Zhang, Y., Lei, J., et al. (2023) · 2023
Closest in time.
The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only
Original
Penedo, G., Malartic, Q., Hesslow, D., Cojocaru, R., Cappelli, A., Alobeidli, H., Pannier, B., Almazrouei, E., and Launay, J. (2023) · 2023
Closest in time.
Yarn: Efficient context window extension of large language models
Original
Peng, B., Quesnelle, J., Fan, H., and Shippole, E. (2023) · 2023
Closest in time.
Understanding optimization of deep learning via jacobian matrix and lipschitz constant
Original
Qi, X., Wang, J., and Zhang, L. (2023) · 2023
Closest in time.
Code llama: Open foundation models for code
Original
Rozière, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Remez, T., Rapin, J., et al. (2023) · 2023
Closest in time.
Tree-based representation and generation of natural and mathematical language
Original
Scarlatos, A. and Lan, A. (2023) · 2023
Closest in time.
Toolformer: Language models can teach themselves to use tools
Original
Schick, T., Dwivedi-Yu, J., Dessì, R., Raileanu, R., Lomeli, M., Zettlemoyer, L., Cancedda, N., and Scialom, T. (2023) · 2023
Closest in time.
Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface
Original
Shen, Y., Song, K., Tan, X., Li, D., Lu, W., and Zhuang, Y. (2023) · 2023
Closest in time.
Reflexion: Language agents with verbal reinforcement learning
Original
Shinn, N., Cassano, F., Labash, B., Gopinath, A., Narasimhan, K., and Yao, S. (2023) · 2023
Closest in time.
Scieval: A multi-level large language model evaluation benchmark for scientific research
Original
Sun, L., Han, Y., Zhao, Z., Ma, D., Shen, Z., Chen, B., Chen, L., and Yu, K. (2023) · 2023
Closest in time.
Stanford alpaca: An instruction-following llama model
Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. (2023) · 2023
Closest in time.
Redpajama: An open source recipe to reproduce llama training dataset
TogetherAI (2023) · 2023
Closest in time.
Llm-powered autonomous agents
Weng, L. (2023) · 2023
Closest in time.
The rise and potential of large language model based agents: A survey
Original
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2023) · 2023
Closest in time.
Effective long-context scaling of foundation models
Original
Xiong, W., Liu, J., Molybog, I., Zhang, H., Bhargava, P., Hou, R., Martin, L., Rungta, R., Sankararaman, K. A., Oguz, B., et al. (2023) · 2023
Closest in time.
Large-scale domain-specific pretraining for biomedical vision-language processing
Original
Zhang, S., Xu, Y., Usuyama, N., Bagga, J., Tinn, R., Preston, S., Rao, R., Wei, M., Valluri, N., Wong, C., et al. (2023) · 2023
Closest in time.
Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x
Original
Zheng, Q., Xia, X., Zou, X., Dong, Y., Wang, S., Xue, Y., Wang, Z., Shen, L., Wang, A., Li, Y., et al. (2023) · 2023
Closest in time.
Gpt-pinn: Generative pre-trained physics-informed neural networks toward non-intrusive meta-learning of parametric pdes
Chen, Y. and Koohy, S. (2024) · 2024
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…